I ran systemctl stop and the service stopped. A second later it was running again. Stopped it again, same thing. I had a unit that treated my stop command as a suggestion, and it took me longer than I'd like to admit to work out why.
The first instinct is always to blame the Restart= policy, and that was half of it. The unit had Restart=always, which does exactly what it says: if the main process goes away, bring it back. But systemctl stop is meant to be exempt from that. A clean stop sends SIGTERM, the process exits, and systemd knows it was a deliberate stop and doesn't restart. So Restart=always alone shouldn't fight a stop. Something else was going on.
systemctl status told the real story once I read it properly:
Active: activating (auto-restart) (Result: exit-code)
Result: exit-code. The process wasn't exiting cleanly on SIGTERM; it was catching the signal, doing some teardown that itself failed, and exiting non-zero. To systemd that looks like a crash, not a clean stop, and Restart=always happily restarts after a crash. So every time I stopped it, it "failed" its way back to life. The fix on that side was making the shutdown path actually exit zero on a normal SIGTERM, plus a sane TimeoutStopSec so a hung teardown gets SIGKILLed rather than hanging about.
That got me most of the way, but it still came back once more, and that was the bit I hadn't spotted. There was a companion .socket unit. Socket activation means systemd is listening on the port, and the first connection starts the service. I'd stop the service, something would connect, and systemd would dutifully start it again on my behalf. Of course it would; that's the entire point of socket activation. I was stopping the service and leaving the thing whose job is to start the service running.
systemctl stop my-app.socket my-app.service
Stop the socket too, and it stayed dead.
The lesson is an old one dressed up in new clothes. When a thing won't stay stopped, you don't have one problem, you have a supervisor doing precisely what you configured it to do. Read the Result: line, check for a .socket sibling, and remember that systemd is not being difficult. It's being obedient. I just gave it two different instructions to keep the thing alive and then got cross when it followed both.