Ramblings of an aging IT geek
← Ramblings of an aging IT geek
linux

the service that wouldn't stay stopped

A systemd service kept respawning seconds after I stopped it, and the answer was socket activation quietly bringing it back to life behind my back.

A Linux terminal showing systemctl status output

I needed a service stopped so I could reconfigure it. I ran the obvious thing.

systemctl stop myapp.service

It stopped. For about four seconds. Then it was running again, with a fresh PID and a start time of just now. So I stopped it again, watched, and again it rose from the dead. At this point you start to wonder if something is wrong with you rather than the machine, which is the correct emotional response and the wrong technical one.

If a unit you've stopped comes back without you starting it, something else is starting it. The trick is finding out who.

First suspect is always a restart policy, so I checked the unit.

$ systemctl cat myapp.service
...
[Service]
Restart=on-failure
...

Restart=on-failure only fires when the process exits non-zero, and a clean systemctl stop is not a failure, so that wasn't it. The unit was being started from outside, not restarted from within. Time to look at what stop actually leaves behind.

A diagram of a socket unit triggering a service

The giveaway was in the journal. Each resurrection was preceded by a log line about an incoming connection on a socket. The application had a companion myapp.socket unit, and socket activation was switched on. That changes the whole picture. With socket activation, systemd itself holds the listening socket open. The service doesn't need to be running to be reachable. The moment any client connects, systemd starts the service to handle it. So my own monitoring, which was helpfully health-checking the port every few seconds, was knocking on the door and waking the thing up the instant I put it to sleep.

The fix is to stop both halves, the socket and the service, together.

systemctl stop myapp.socket myapp.service

Stop the socket first and systemd stops listening, so the next health check gets a connection refused instead of triggering a start. Then the service stays down. If you want it to stay down across a reboot too, disable or mask the socket as well, not just the service, otherwise the socket comes back at boot and the whole cycle starts over.

systemctl disable --now myapp.socket

The lesson I took away: when you stop a service, ask what else can start it. Socket activation is genuinely good (it gives you lazy startup and clean ordering for free) but it quietly moves the "who's listening" responsibility from the service to systemd, and if you forget that, you'll spend an afternoon fighting a process that is only doing exactly what you told the system to do. It wasn't refusing to die. I just kept ringing the doorbell.