Let me be clear at the outset about the return on investment here: there wasn't one. I built a four-node Raspberry Pi cluster, ran Kubernetes on it, and at no point did it do anything I couldn't have done better on a single old laptop or a cheap VPS. It was, by any sensible measure, a waste of a weekend and the better part of two hundred quid. I'd do it again tomorrow.
The pitch I told myself was that I'd "learn Kubernetes properly" by running it on real, separate machines instead of in a single virtual cluster on my desktop. There's a sliver of truth in that. Mostly I just wanted to build a small stack of blinking computers, and a tidy justification helps the purchase go through.
The build
Four Raspberry Pi 4s, the 4GB models, in a stacked acrylic case with little brass standoffs and a pair of fans that turned out to be louder than the rest of the house combined. PoE would have been the elegant choice; I went with four USB-C supplies and a short power strip, which is the inelegant choice that actually arrived next day.
Networking was a cheap eight-port gigabit switch. Storage was microSD cards, which everyone warns you against, and they're right, but I wanted the thing running this weekend, not after a NAS arrived. I flashed each card with a 64-bit OS, set hostnames pi0 through pi3, dropped my SSH key in, and disabled the password login.
The first real lesson arrived before any software: cable management on something this small is a genuine problem. Four power leads, four network cables, and the fan wiring turned a neat stack into a hedgehog. I spent an embarrassing amount of time with a bag of zip ties making it look like I knew what I was doing.
Kubernetes, the lightweight kind
Full Kubernetes on a Pi 4 is asking a lot. I went with k3s, which strips out a pile of the heavier components and ships as a single binary. The control plane went on pi0 and the other three joined as agents.
# on pi0, the server
curl -sfL https://get.k3s.io | sh -
# grab the join token
sudo cat /var/lib/rancher/k3s/server/node-token
# on pi1..pi3, the agents
curl -sfL https://get.k3s.io | \
K3S_URL=https://pi0:6443 \
K3S_TOKEN=<token> sh -
A few minutes later, kubectl get nodes showed four boxes in Ready, and I will admit to a small, disproportionate thrill at that. Four physical computers, sat on the desk, all agreeing they were one cluster. There's something about it that a virtualised setup never gives you, even though the virtualised setup is in every practical sense superior.
What I actually ran on it
This is where the project's lack of purpose became undeniable. I deployed a handful of things to give it a reason to exist: a self-hosted dashboard, a small monitoring stack with Prometheus and Grafana, and a couple of toy services I wrote to generate load. The monitoring stack promptly ate most of the cluster's memory, which was itself instructive: half the resources of my "cluster" went to watching the cluster.
I poked at failure. Pulling a node's power and watching the workloads reschedule onto the survivors is genuinely satisfying the first time. Watching a microSD card corrupt itself under sustained writes a fortnight later was less satisfying, and entirely predictable. Lesson noted, NAS now on the actual shopping list.
Did I learn anything?
The honest answer is: not much I couldn't have learned cheaper. The mechanics of k3s, sure. A bit about ARM builds and multi-arch images, which bit me when half my container images simply didn't exist for ARM. A real, in-the-fingers feel for what "the node went away" looks like rather than the abstract idea of it. But nothing that justified the spend or the fan noise.
And yet. There's a kind of learning that doesn't show up on a CV and doesn't need to. Building something physical, watching it boot, breaking it on purpose, fixing it: that keeps the part of me that got into this for fun alive. Not every project needs to pay rent. Some of them just need to blink reassuringly on the desk and remind you why you liked computers in the first place. This one does exactly that, and I've stopped apologising for it.