Ramblings of an aging IT geek
← Ramblings of an aging IT geek
hardware

a raspberry pi cluster that taught me nothing useful, and i'd do it again

Building a four-node Raspberry Pi cluster that solved no real problem, ran nothing important, and was the most fun I'd had with hardware in ages.

A soldering iron and electronics on a bench

I built a four-node Raspberry Pi cluster this autumn. It does nothing useful. It runs no workload that wouldn't be happier on the single Intel NUC sitting two shelves up, drawing less power and giving me less grief. I knew all of this before I started. I built it anyway, and it was the most fun I've had with hardware in a year, so this is the unapologetic write-up of a project with no business case.

the bill of materials, and the first lie I told myself

Four Raspberry Pi 3 Model B+ boards, a stackable acrylic case with little brass standoffs, four microSD cards, a short run of flat ethernet to a cheap five-port gigabit switch, and a multi-port USB power supply so I wasn't running four separate wall warts. The first lie was "this'll be cheap". By the time I'd added the case, the switch, decent SD cards rather than the bargain-bin ones that corrupt themselves, and a USB PSU that could actually deliver the current four Pis want under load, I was well past the cost of a single far more capable machine. The Pi is cheap. The Pi cluster is a hobby with a Pi-shaped entry fee.

The build itself is genuinely lovely though. There's something deeply satisfying about the physicality of it: four identical boards, standoffs screwed down, the little stack growing on the desk, cable management that I told myself I'd do properly and mostly did. It scratches the same itch as Lego, and I'm not going to pretend that isn't a large part of the appeal.

A circuit board close-up

flashing four cards is four times the tedium

Here's the first thing the project actually teaches, and it's not about computing: doing anything four times by hand is four times as annoying as doing it once, and you will want automation immediately. Flashing four SD cards, setting four hostnames, enabling SSH on four boards, that's where the romance wears thin fast.

I did the first one by hand, then wrote myself a small script to stamp out the rest. Mount the boot partition, drop in an empty ssh file to enable SSH on first boot, write a hostname, set a static-ish address. Nothing clever:

#!/usr/bin/env bash
set -euo pipefail
BOOT="$1"      # path to the mounted boot partition
HOST="$2"      # e.g. pi-node-2
touch "$BOOT/ssh"
echo "$HOST" > "$BOOT/hostname-stamp"
echo "stamped $HOST onto $BOOT"

It's barely a script. But it turned a fiddly per-board ritual into something I could repeat without thinking, and the lesson there, treat the fleet as cattle from the very first node, is the one genuinely transferable thing in the whole project. Even at a scale of four.

then what do you actually run on it

This is where the "taught me nothing useful" claim earns its keep. The honest answer to "what do you run on a four-node Pi cluster" is "whatever lets you justify having built it". I went through the predictable stations of the cross:

  • A Kubernetes distribution, which the Pi 3's gigabyte of RAM and that famously shared USB-and-ethernet bus resent enormously.
  • A toy distributed task queue, which worked, and which a single Pi would also have run perfectly well.
  • sysbench and assorted benchmarks, purely to watch the numbers, which is a confession not a methodology.

The cluster orchestration stuff "works" in the sense that pods schedule and pods run, but the moment you put any real I/O through a Pi 3 you remember that the ethernet shares a USB 2.0 bus with everything else, and the SD card is the slowest storage you've owned since about 2006. So it taught me what the Pi 3 is bad at, which I already knew from the spec sheet, which is why I'm comfortable saying it taught me nothing useful.

What it did do is make distributed-systems concepts physical. There is a difference between reading about a node going away and reaching over to pull the power on node 3 to watch the scheduler shrug and reschedule. The cluster being slow and small and right there on the desk made the abstractions tangible in a way a cloud console never quite does. I could see which board's activity LED was thrashing. I could feel which one was warm.

the actual lesson is about motivation

If you came to this for a recommendation, here it is, and it's the opposite of what the build photos suggest. If you want compute at home, buy one capable machine. A NUC or a tidy second-hand small-form-factor desktop will out-perform a Pi cluster on every axis that matters, cost it included, and you'll spend your evenings using it rather than nursing four SD cards.

But if the building is the point, if you want to hold a distributed system in your hands and break it on purpose and watch it heal, then a small Pi cluster is a wonderful, slightly absurd toy. Just be honest with yourself about which of those two people you are before you order the standoffs.

I was clearly the second. The cluster is still on the desk. It still does nothing important. I look at the little stack of boards and the tidy cabling and I'm pleased every single time, and at the price of admission to a hobby, that's a perfectly good return.