January 18, 2017

Building a 144 core Raspberry PI 3 Cluster

A couple of months ago, we built a cluster of 36 Raspberry PI 3 (4 cores each, so 144 cores in total), resulting in 2304 GB of storage and 36 GB of RAM. There are many tutorials about building small Raspberry PI clusters available online, but hardly any of them cover what you can do with them in a business environment. So, why did we spend a lot of time to get this thing to work?

144 Core Raspberry PI 3 Cluster
First, our cluster is like a data center in a box. It just has a network cable and a power cable, so it actually is “plug and play.” Everything else is contained in the cluster shown in the picture. We had to build the hardware, the electronics, and to create the networking, routing, etc. We asked a team of software engineers and operations specialists to work closely together on this project. Participating in a project like this gives each of the members insight into other areas that are usually not part of their daily work; software guys learned more about networking and how to set up the proper administration of servers, and operations guys learned more about the software side.



Another very important point is that this mini data center is a playground. We can try out new things without any risk, and at low cost. What if we want to test a service with 20 instances running on multiple hosts? No problem - it’s up and running in a few minutes. No starting of VMs, no starting of any cloud instances that charge by the minute – we can just go and try it out. This gives us the chance to try out new things, play around, and mess things up without any risk. For us, this is really important, as we always want to be able to move forward very quickly. Each month, we spend an entire day building stuff that we engineers want to build and find useful. Some of it needs to be hosted somewhere, and these days, we want to achieve things quickly. Our mini data center is an ideal place to try out and house our ideas on these days (we call them “Crazy Fridays,” even though they held are on a Thursday).

There is another very practical aspect to having a mini data center in the office. Big data, scalable architectures, resilient software design, automatic fail over, and leader election – all of these areas affect software, and that software needs to be tested to respond to cases of errors and other circumstances, like network failures. Have you ever tried to get into a data center and plug in a cable? Slowed down a switch on purpose? I think you can maybe do it once, before you’re put on the blacklist of people not allowed to be there. In your personal mini data center, however, you are free to do whatever you want and to test your software and algorithms in any way you choose.

Furthermore, we use the cluster to host internal applications, like information screen back-ends. Not reliable enough? Running an application on this cluster with a replication factor of ten provides enough room for errors (OK, so there’s no redundant power supply and so on, but that is out of scope and not needed for these kinds of applications). Hosting your own applications on your own mini data center is just fun.

Last but not least, there was a lot of hardware to build, such as the actual blue case shown in the picture and the acrylic glass parts. We used a CNC machine and a laser cutter to get all the parts in place. Building actual, physical things is a lot like building software; there are problems that need to be solved. Also, it broadens the minds of people who spend most of their time just creating virtual things when they are involved in building something they can touch.

If you want to build your own, all plans are provided in our github repository.

So, what do you need to build it?


36 x Raspberry PI 3
36 x 64 GB SD card
36 x CAT6 network cables 1.5m
36 x USB cable (for power supply)
144 x M2.5 screws
144 x M2.5 nuts
6 x High power USB hubs (like this one)
1 x 48 port switch


Building the Hardware

The housing was made of medium-density fiberboard and, of course, you need acrylic glass for the other parts of the tower.

For the bottom housing that contains all the networking and power supplies, we just cut out a lot of rings and glued them together so that we had the shape we wanted.

video



The acrylic glass parts were cut using a laser cutter:
Engraving our logo

Vertical plate holders

Our rings to hold the raspberry PIs


After that, we just had to get the housing painted ...
Painted housing


... and try to fit everything in it ...
Power Supply
48-Port Switch

Network cables
SD Cards


... add some fancy lights ...

video



and you are done! It won't take you more than 20-30 hours.


Nevertheless, to make this cluster really usable and easily accessible for everyone who wants to play around with it, we decided to go for a software setup that enables us to deploy any application with just one line of code, including fail over, scaling, etc. We’ll use Docker Swarm to power it.


This software part will be covered in the next installment of this blog article.

9 comments:

  1. cool! Thanks for sharing this post. I would like to write about it in my custom thesis

    ReplyDelete
  2. Interesting stuff for sure. Looks dope too :)

    ReplyDelete
  3. You have posted a very detail document. I read all of your article and I really like it, I understand your point of view.
    - Google Street View

    ReplyDelete
  4. The productivity of such a data center is significant for everyone. It increases the efficiency of previous functions and increases the speed of work.

    ReplyDelete
  5. We can try out new things without any risk, and at low cost. What if we want to test a service with 20 instances running on multiple hosts? No problem ?

    ReplyDelete
  6. In any case, to make this bunch truly usable and effortlessly open for everybody who needs to play around with it, we chose to go for a product setup that empowers us to convey any application with only one line of code, including bomb over, scaling, and so on. We'll utilize Docker Swarm to power it. Professional Research Writing

    ReplyDelete