I spent an embarrassing amount of an evening trying to stop a service. systemctl stop would return cleanly, systemctl status would show it inactive, and a few seconds later it was running again with a fresh PID, cheerfully serving requests I very much did not want it serving while I was meant to be replacing it.
The first instinct is always a rogue cron job or some other process group reviving it. It was not. The unit was reviving itself, and it took me longer than I would like to admit to spot why, because the answer was sitting in plain sight in two places at once.
The first culprit was Restart=always. Fine in principle, that is why it was there. But the way I was killing it, an over-enthusiastic kill -9 on the main PID while I was poking around, looked exactly like a crash to systemd. So it dutifully did its job and restarted the thing I had just murdered. That is not a bug, that is the entire point of the directive. I was fighting my own configuration.
The second culprit was nastier, and the real reason systemctl stop did not stick: socket activation.
# foo.socket
[Socket]
ListenStream=/run/foo.sock
[Install]
WantedBy=sockets.target
The service had a partner .socket unit. Stopping foo.service left foo.socket happily listening, so the moment anything touched the socket, systemd started the service straight back up to handle the connection. From the outside it looked like the service refusing to die. In reality it was being summoned, on demand, by the socket I had not thought to stop.
The fix, once I understood it, was dull in the way good fixes are. Stop and mask both halves while I worked:
systemctl stop foo.socket foo.service
systemctl mask foo.service
Masking points the unit at /dev/null, so nothing, not socket activation, not a dependency, not me fat-fingering a start, can bring it back until I unmask it. With the socket stopped and the service masked, it finally stayed in the ground.
If you want to see what is actually wired to a unit before you go to war with it, systemctl list-dependencies foo.service and systemctl status foo.socket will both tell you. I knew that. I did not do it, because the service looked simple and I assumed I understood it, and assuming I understood it is precisely what cost me the evening. Five seconds of looking would have shown me the socket sitting there, armed and waiting.
The lesson I keep relearning is that with systemd, a service is rarely just a service. It is a service, possibly a socket, possibly a timer, possibly a path unit, all wired together to do exactly what you told them. When something refuses to stay dead, the question is not "what is restarting it" but "what did I configure to restart it". The machine was doing precisely as instructed. The fault, as ever, was upstream of the keyboard.