I spend my days building distributed systems, so I went into a weekend multiplayer prototype in Unreal feeling, let's say, confident. Two players, a ball, a bit of physics. How hard can it be? I have shipped services that move millions of messages. Reader, it was humbling, and the humbling came from exactly the place my day job should have prepared me for.
The first lesson is that Unreal's replication is not the same thing as "sending data over the network", and treating it like a message bus will hurt you. You don't push updates. You mark a property as replicated, the engine decides when and whether to send it, and the client receives it whenever it gets round to. I kept reaching for "send this now" and the framework kept gently refusing. It has its own model of authority and relevance, and you work within it or you fight it the whole way.
The lesson that actually drew blood was authority. Early on I let the client move the ball and tell the server about it, because that felt responsive. Of course it did. It also meant the client was the source of truth, which is the cardinal sin. The moment I had two clients with slightly different ideas of where the ball was, there was no answer to "but where is it", because nothing was authoritative. The server has to own the simulation. The client predicts and corrects. That distinction is the whole game, and I'd waved it away because it felt like ceremony.
What made me wince is that this is the same lesson as every distributed system I've ever debugged. One writer, or you're going to have a conflict you can't resolve. I knew that. I have given that exact talk to junior engineers. And I still walked straight into it the moment the domain was unfamiliar and the feedback was a ball jittering across the screen instead of a graph going wrong.
There's a particular humility in watching a RepNotify fire in the wrong order and realising your beautiful mental model of "the state is the state" is a lie the network tells you. Eventual consistency feels abstract until it's a physics object teleporting because two updates arrived out of sequence. I've read about it for years. Feeling it in my hands, at 1am, chasing a rubber-banding ball, taught me more than the reading ever did.
The prototype is rubbish, by the way. Two cubes and a ball, no art, no polish. But I came away with more respect for the people who ship netcode than I've had for any subfield in a long while. They live, full-time, in the place my distributed-systems brain only visits when something is already on fire. Humbling is the right word.