Part 3 of 5 - follows on from; Part 2 - home computing journey

My desire for our home server / lab, was to have various workloads (apps) NOT bound to specific physical server(s). In essence I wanted for my home the same high-availability that is available to enterprise applications. Maintenance on any one physical server (node) or disk drive to have little effect on the applications running on the cluster.

To achieve this, the nodes in the cluster need to be networked in such a way that they could all agree, forming consensus, as to the state of the cluster, and on which node an application was currently running. Should that node fail, the application would be immediately restarted on another node the cluster deemed to be available.

This in turn requires a minimum of three servers, and ideally an odd number at that, otherwise a situation might develop called a split-brain. The simplest way to explain this is if one were to have a two-node cluster, merely the failure of the network link would cause both sides to be unable to determine if the other was available, and both sides might try and run the application. This has particularly bad results for database applications.

So, a three-node cluster with redundant network links can survive a single network failure.

This principle also applies to distributed storage. Each of my servers was to house a several disk drives for the storage cluster. Data, written in blocks to the pool, are mirrored to another node. Thus, a block of data exists on two seperate servers, and a failure of a single node might remove at most one copy of that block of data from being available to running application.

The Ceph storage system, autonomically, will recreate a missing block on an another node, if available, after a certain period of time, maintaining the prescribed 2-way mirroring of data.

What cannot be autonomically recreated is a new server. Should one of my three servers blow up, requiring a physical replacement, this would require ordering parts and reinstalling etc. During this period, the cluster is in a vulnerable state, being capable of entering a split-brain scenario if the surviving pair fail to agree on cluster consensus. Additionally the Ceph storage system starts to struggle with a very small number of nodes.

In order to mitigate this, a solution is to run a five-node cluster. Such a cluster can easily sustain two servers being offline at any one time. Furthermore the number of disks in the Ceph storage pool would then perhaps be between ten and twenty, which provides for a quite a performant system, even utilising older SSDs.

Procurement

Looked to buying low power Intel Atom 3000 series servers, the Supermicro SYS-5019A-FTN4 with an embedded 8-core Atom CPU stood out as a contender. In 2020, they sold for AUD2,950 each.

The Supermicro Included

  • 4x gigabit ethernet network interfaces
  • 4x SFF internally disk slots
  • 1x M.2 SSD disk slot
  • 1x PCIe interface card slot
  • 1x Atom C3758, 8-Core, TDP 25W CPU
  • no memory (4 slots)
  • 200W power supply

They required (to complete the cluster)

  • 10Gbe PCIe network card (faster networking)
  • 2-4x 2½” SFF disks (for Ceph storage pool)
  • 1x M.2 disk (for system volume)
  • 64GB memory (4x 16GB RDIMM)

This would have provided 4x gigabit ethernet, and support for (but yet to be provided) 4x 2½” SFF (small form factor) disks, 4x ECC DIMM slots for memory, and 1x PCIe slot for an add-in card for 10Gb networking. The Atom server would not have provided redundant power supplies.

All up, without Ceph disks, probably about AUD4,500 each. That’s eighteen grand for four servers, without the storage pool disks.

Compared to the 2013 era HPE SL250s Gen8 that I found on eBay.

Which included

  • 2x gigabit ethernet
  • 4x SFF disk front-mounted (w/- additional internal 4x SFF disk)
  • 1x 10Gb 2-port 544FLR-QSFP Adapter
  • 3x GPU slots (in-place of installed internal 4x SFF disk)
  • 2x Xeon E5-2620, 6-Core, 95W (ea.) CPU
  • 64GB ECC memory (as 8x 8GB 12800R DIMMS)
  • shared redundant power with HPE SL6500 chassis
  • 4x 1TB SATA SFF disks

Required (to complete cluster)

  • 2-4 2½” SFF disks for Ceph storage pool.
  • disks for system volume

The seller on eBay had three of these HPE SL6500’s available. Each with four SL250s servers. The differences being CPU speed and included memory. The low end SL6500 only had 32GB RAM and 2x 1TB SATA SFF disks in each server.

So for AUD4,000, I had 4x HPE SL250s as outlined above. Only to add the disks, for which I already had 3x Intel PCIe SSDs for the boot/system disks.

For the storage pool disks, I had been keeping an eye on the value curve for 2-8TB mid-range enterprise SSDs, particularly the Micron 5300 PRO series. Along came a wholesale offer for a batch of older, but unused, 5100 MAX, at a steep discount to new SSDs of similar performance and endurance. I settled on a larger number of smaller disks, filling the front-load SFF slots.

So for another AUD5,000, I had 20x 2TB SATA SSDs

All up, AUD 9,000 probably seems like a lot for a home server, and it is. It is also less than a boat or a fancy push-bike too. Both of which might be someone’s passion/hobby.

The fifth server, certainly not a fifth wheel, was to be my existing Supermicro server, then eBay jumped out at me with another HPE Gen8 server. This was a piece of metal I have always thought was kinda cool. The mighty DL580 not a fifth wheel

Part 1 - What is MiniPart 2 - home computing journeyPart 3 - converged cluster (this post)Part 4 - not a fifth wheelPart 5 - our Mini computer