Ramblings of an aging IT geek
← Ramblings of an aging IT geek
golang

cross-compiling go for an arm box, the long version

A fuller account of cross-compiling Go for ARM, including where cgo, the GOARM flag and static linking actually bite.

A close-up of source code on a screen

I wrote a short note last week saying cross-compiling Go for an ARM box was almost free, and for the happy case it genuinely is. Then I tried to do it for real, with a program that wasn't a toy, and discovered all the places where "almost free" quietly turns into an afternoon. This is the fuller account, written down mostly so future me stops relearning it.

The target is a small ARMv6 board running Linux, the kind of thing you leave in a cupboard. My laptop is x86-64. The goal is to never compile on the board itself, because it's slow and I'd rather not install a toolchain on a box whose whole appeal is that it does one thing.

the happy path, properly

For a pure-Go program with no C dependencies, the whole exercise is three environment variables:

GOOS=linux GOARCH=arm GOARM=6 go build -o thing-arm ./cmd/thing

GOOS and GOARCH are obvious. GOARM is the one people forget, and it's the one that bites quietly. It selects the floating-point ABI and instruction set: 5 for the very old soft-float chips, 6 for the original Pi, 7 for anything Cortex-A and newer. Get it wrong in the "too new" direction and the binary either won't run or will trap on an illegal instruction the first time it touches a float. Get it wrong in the "too old" direction and it runs, just slower than it needed to. The failure mode where it works on your test box and falls over on the deployment box is the worst kind, because you'll swear blind the binary is fine.

Verify what you built before you copy it anywhere:

file thing-arm
# thing-arm: ELF 32-bit LSB executable, ARM, EABI5 ...

If file says ARM, you're in business. If it says x86-64, you forgot to export the variables and built for your own machine, which I have done more times than I'll admit.

A wider shot of code on a monitor

then cgo turns up and ruins everything

The moment your program imports a package that uses cgo, the free ride ends. The usual culprits are anything touching SQLite, certain crypto bindings, or os/user and net in their cgo-backed resolver modes. Go will happily compile pure-Go packages for a foreign architecture, but it cannot compile C for ARM without a C cross-compiler, because it doesn't ship one.

You have two honest choices.

The first is to avoid cgo entirely. For net and os/user, you can force the pure-Go implementations:

CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o thing-arm ./cmd/thing

CGO_ENABLED=0 does two useful things at once: it forces the pure-Go code paths, and it produces a statically linked binary with no libc dependency at all. That last point matters more than it sounds. A static binary doesn't care what version of glibc the target has, or whether it has glibc at all, which means the same file runs on the Pi, on an Alpine-based container, and on whatever minimal image I reach for next. For small network tools this is almost always what I want.

The second choice, when you genuinely need a C library, is to install a real cross toolchain and point Go at it:

CC=arm-linux-gnueabihf-gcc \
CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=6 \
go build -o thing-arm ./cmd/thing

Now you need the matching gcc-arm-linux-gnueabihf package installed, and if the C library you're linking has its own dependencies you may need a sysroot with those headers and libs too. This works, but it's exactly the faff I was trying to avoid, and it reintroduces the glibc-version coupling that the static build made disappear. I treat needing this as a signal to ask whether I really need that dependency.

a word on reproducibility

There's a quieter benefit to building these binaries on my laptop rather than on the board, and it's that the build environment stops being a mystery. The board accumulates state the way any long-lived box does: a Go version I bumped once and forgot, an environment variable set in some shell profile, a package installed to debug something two years ago. Build there and you're never quite sure what went into the result. Build on the laptop, from a checkout, with the variables stated explicitly on the command line, and the inputs are all visible in front of you.

I lean on that by keeping the build invocation in a tiny shell script in the repo rather than typing it from memory each time. It's three lines and it never drifts, which means the binary I ship today is built the same way as the one I shipped last month. That sounds like a small thing until the day a build behaves differently and you have to work out what changed, at which point "nothing in the build command, because it's checked in" is a wonderful answer to be able to give.

the workflow I actually use

Once the build is sorted, the rest is unglamorous and that's the point:

CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -o thing-arm ./cmd/thing
scp thing-arm pi:/usr/local/bin/thing
ssh pi 'systemctl restart thing'

Build, copy, restart. No compiler on the target, no shared-library surprises, no waiting twenty minutes for the board to grind through a build it shouldn't have to do. The whole loop is a few seconds, most of which is the scp.

what I'd tell myself

Keep these tools pure Go on purpose. The temptation to pull in a convenient C-backed library is real, and every time you give in you trade a one-line build for a toolchain, a sysroot, and a coupling to the target's libc. For a small daemon that lives in a cupboard, none of that is worth it.

Set GOARM explicitly and write it down somewhere, because the day you forget it is the day the binary works perfectly on your desk and fails only in the cupboard. And run file on the output every single time, because the most embarrassing bug in this whole process is shipping an x86 binary to an ARM box and spending ten minutes wondering why it won't execute.

It is, still, one of the genuinely lovely things about Go. Most languages make cross-compilation a project in itself. Here it's three variables and a value judgement about cgo, and once you've internalised where the edges are, it really is almost free.