The thing nobody tells you about rootless containers is that the runtime is the easy part. Podman rootless has been fine for ages. What gets you is everything that quietly assumed it was root: bind mounts owned by your user, ports below 1024, and the cgroup delegation that systemd may or may not have set up for you.
Two settings fixed most of my grief. First, /etc/subuid and /etc/subgid actually needing a sane range for the user, otherwise chown inside the container fails with a baffling operation not permitted and you spend an hour blaming the image. Second, the privileged-port problem, which I solved by lowering net.ipv4.ip_unprivileged_port_start rather than reaching for capabilities I didn't want to reason about.
The payoff is real, though. A compromised container is now a compromised unprivileged user, not a compromised host. I migrated three services this weekend with podman generate systemd for the units, and nothing has needed root since. The mental model is cleaner too: if a thing wants root, I now have to justify it out loud, and most of the time the answer is that it doesn't.
Was it worth a Sunday? Yes, mostly because I sleep slightly better. Not a glamorous win, but the best ops wins rarely are.