The bug looked like a netcode ghost story. Players would occasionally not register a hit that very clearly landed, but only under load, only on a busy server, and never in the editor where I could watch it. Everyone's first instinct was packet loss. It was not packet loss. It was me misunderstanding the word "reliable."
In Unreal, an RPC marked Reliable does not mean "this will always arrive." It means "this will arrive in order, and the engine will retransmit it." There is a queue behind that promise, and the queue is not infinite. Flood the reliable channel faster than it can drain, and the engine does not block, it does not panic, it just starts quietly dropping things to keep the connection alive. The footgun is that the word on the tin tells you the opposite of what happens when you abuse it.
UFUNCTION(Server, Reliable)
void ServerReportHit(const FHitData& Hit);
That signature is fine for an event that happens a few times a second. We were firing it per-projectile, per-frame, on full-auto weapons, for every client at once. The reliable buffer filled, and the engine started discarding RPCs to protect ordering on the ones it kept. Hence: hits that "landed" but never reported, only when enough was going on to saturate the channel.
The fix was less about a flag and more about respecting the channel as a scarce resource. A few things, in rough order of how much they helped.
- Batch. Instead of one reliable RPC per hit, accumulate hits for a tick and send them as a single array. One RPC carrying ten results costs the queue a lot less than ten RPCs.
- Demote what can be demoted. Cosmetic feedback (a muzzle flash, a tracer) does not need reliability at all. Make it
Unreliableand let the occasional one vanish. Nobody notices a dropped tracer; everybody notices a dropped kill. - Stop sending what the server already knows. A lot of our reliable traffic was the client telling the server things the server could derive itself. Authority should compute, not be informed.
The deeper lesson is that "reliable" in a game networking layer is a budget, not a guarantee you get for free. The engine will keep your connection healthy at the cost of the messages you carelessly overcommitted to it, and it will do so silently, which is the worst way for it to do anything. Once we treated the reliable channel like the limited thing it is, the phantom misses went away. The hits were always landing. The reports just never made it out of the queue.