A backup you have never restored is not a backup. It is a hopeful file. I knew this in the abstract for years, the way everyone does, right up until the evening I needed to restore one and discovered the archive had been silently truncated for months.
Nothing was lost in the end, because the data also lived somewhere else, which is luck rather than strategy. But it rearranged how I think about the whole thing. The question stopped being "am I backing up" and became "when did I last prove I could get it back".
What changed
I moved everything onto restic. The deduplication is genuinely lovely, the encryption is on by default, and the repository format has held up under the kind of poking that breaks naive tar-and-rsync setups. A typical job is unremarkable, which is exactly what you want from a backup tool:
restic backup /srv /etc \
--exclude-file /etc/restic/excludes.txt \
--tag nightly
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
The part that actually matters is not the backup command, though. It is the restore drill. On the first Sunday of the month a script picks a random snapshot, restores it into a scratch directory, and checks a handful of known files are present and the right size. If it passes, I get a quiet line in the log. If it fails, I get an email I cannot ignore.
A few things I now treat as non-negotiable:
restic check --read-data-subsetruns weekly, because a repository that says it is healthy and a repository that can actually read its own data are not the same claim.- At least one copy lives off-site, on a box I do not administer day to day, so a mistake here does not propagate there.
- The restore drill targets a different machine to the one being backed up, so I am testing the path I would actually use in a disaster, not a convenient shortcut.
None of this is clever. That is the point. The clever part was believing, for years, that a green cron job meant my data was safe. It meant the job ran. Whether the result was any good was a separate question I had simply never asked.
The reassuring thing is how dull it has become. Once a month the drill runs, restores something at random, tells me it worked, and I get on with my evening. The first time it told me it had worked, I almost did not believe it. Now it is just background noise, which is the highest compliment I can pay any piece of infrastructure.