I wanted to take a service down for a few minutes to swap a config file. Simple enough. systemctl stop myapp, edit the file, start it again. Except it wouldn't stay stopped. I'd stop it, systemctl status would confirm it was inactive, and within a few seconds it was back, running, cheerful, as if I'd never touched it.
My first thought was Restart=always in the unit, the obvious culprit. But that only restarts a service that exits unexpectedly, and I was stopping it cleanly, which systemd treats differently. The status output even agreed with me: it said inactive, then a moment later said active again, with no crash in between.
The answer was sitting in a file I'd written months earlier and forgotten. There was a companion myapp.socket unit, and socket activation was enabled. systemd was holding the listening port open on the service's behalf, and the instant anything connected, it dutifully started myapp.service to handle the request. Stopping the service did nothing to the socket. The socket was still listening, something reconnected, and up it came again.
$ systemctl stop myapp.service myapp.socket
Stop the socket as well and it finally stayed down. To do the config swap properly I masked neither and just stopped both, edited, started both. Obvious in hindsight, which is the most annoying kind of obvious.
The lesson I keep relearning with systemd is that a service is often not a single thing. It's a service unit and maybe a socket and maybe a timer and maybe a path watcher, all quietly cooperating. When one half refuses to die, check what's keeping it warm.