I spent the best part of a day convinced I'd found a bug in Unreal's networking. I had not. I'd found a bug in my own assumptions, which is the more common species and a good deal more embarrassing.
The setup was a small co-op prototype. A player triggers an ability, the server validates it and tells the relevant clients to play an effect. Standard server-authoritative stuff. I wired it up with RPCs, marked the cosmetic ones Unreliable because they're just effects, and marked the important state changes Reliable. Then I watched effects occasionally fire in the wrong order, and occasionally not fire at all, and I started muttering about engine bugs.
what reliable actually promises
The first footgun is the word. A Reliable RPC in Unreal guarantees delivery and guarantees ordering relative to other reliable RPCs on the same channel. It does not guarantee anything about timing relative to Unreliable calls, and it does not promise the call survives an actor going out of relevance. That last one bit me hard. If the target actor isn't relevant to a client (out of network range, culled, whatever), the RPC simply doesn't get sent to them. Not delayed. Not queued. Gone.
UFUNCTION(Server, Reliable, WithValidation)
void ServerActivateAbility(int32 AbilityId);
UFUNCTION(NetMulticast, Unreliable)
void MulticastPlayEffect(FVector Location);
So my "important" reliable call would land, the server would do its thing, and then a multicast effect would fire for some clients and not others depending entirely on relevance, which I'd never reasoned about because everything looked fine when two players stood next to each other.
the bit that actually hurt
The ordering surprise was subtler. Reliable and unreliable RPCs travel on different paths, so a reliable state change and an unreliable effect that I'd written on adjacent lines could arrive in either order on the client. I'd assumed code order implied wire order. It does not. For cosmetic effects that's harmless. The moment a piece of gameplay logic depended on an unreliable call having arrived, I had a race that only showed up under packet loss, which is to say only in front of actual players.
how I stopped fighting the engine
A few rules I've since beaten into my own head. Anything the game state depends on goes through replicated properties with RepNotify, not through RPCs, because property replication is built to converge and RPCs are not. RPCs are for events, not for state. And anything important is Reliable and is driven from a component on a relevant actor, the player state or the player controller, so relevance doesn't quietly eat it.
The cosmetic effects can stay unreliable and unordered, because if a player misses one impact spark under heavy packet loss, nobody dies. The header says reliable, and it is, for the precise thing it promises. The footgun is reading more into that word than the engine ever offered. Replicate state, fire events, and don't let an effect carry a fact that the simulation needs to be true.