I wanted my containers to run as me, not as root, on a box where I don't fully trust everything I'm pulling. Rootless Podman is the right tool. Getting there cost me an afternoon, almost all of it spent on two things nobody mentions until you hit them.
First, subuid and subgid. Rootless containers map the container's UIDs into a range allocated to your user, and if that range isn't set up you get a single uid and nothing else works. Check /etc/subuid and /etc/subgid have a line for you, something like johnm:100000:65536. If not, usermod --add-subuids 100000-165535 --add-subgids 100000-165535 johnm and then podman system migrate to pick it up.
Second, bind mounts and permissions. A volume mounted from your home directory will appear inside the container owned by some absurd high-numbered uid, because that's the mapping. Writing back to it fails in confusing ways. The fix is :U on the mount, which tells Podman to chown the volume contents to match the mapped user, or use a named volume and stop fighting it.
After that it just works, including systemctl --user units generated with podman generate systemd, which survive reboots once you loginctl enable-linger. No daemon running as root, no socket to worry about. Quietly excellent, once you've paid the entry fee.