I run almost everything rootless now, and for a while that meant a low background hum of permission errors. Bind mounts owned by the wrong user, services that worked under root and died as my unprivileged user, the lot. The frustrating part was that none of it was Podman's fault. It was me not understanding what user namespaces were actually doing to my files.
Two things fixed most of it. First, the subuid and subgid ranges. If /etc/subuid and /etc/subgid don't have a decent allocation for your user, rootless containers can't map enough IDs and you get inscrutable failures deep inside an image build. One line each, 65536 IDs, sorted:
johnm:100000:65536
Second, --userns=keep-id for the containers that bind-mount my actual home directory. Without it, the container's "you" maps to some high subuid on the host, so files it writes land owned by a UID that doesn't exist when you ls them. With it, the inside-and-outside UID match and the ownership just looks normal.
The thing nobody tells you is that rootless isn't harder than rootful, it's differently shaped. Once I stopped reaching for sudo out of habit and actually read what the namespace was mapping, the whole setup got calmer. It's been weeks without a single permission tantrum now, which for me is a personal best.