ResearchPapers, evidence & method
Ai2 releases StateBench; frontier models score 13% or lower on its hardest long-document state-tracking tasks
Ai2's StateBench grades how well language models follow changing variables across long documents. Frontier models handle simple lookups and latest-value updates but collapse on chained, reference-based updates, while small recurrent and hybrid models trained on the tasks generalize far beyond training length.
From the Hugging Face article of November 3, 2026 and cited sources: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna did well on immutable-assignment and reassignment-by-value problems. [1] [5]
SOURCES[1] StateBench: Testing how language models track state… huggingface.co[5] A Fine-Grained Hierarchy and Benchmark for State Tracking… arxiv.org
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna did well on immutable-assignment and reassignment-by-value problems. [1] [5]
- Original source
- Hugging Face article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 3, 2026 article published
- Covered
- Nov 5 edition
- Evidence
- 10 sources across 8 sites
- 1StateBench: Testing how language models track state…huggingface.co
- 2The Illusion of State in State-Space Modelsarxiv.org
- 3Unlocking State-Tracking in Linear RNNs Through Negative…arxiv.org
- 4OLMo Hybrid Architecture Achieves 2x Data Efficiencyai2.work
- 5A Fine-Grained Hierarchy and Benchmark for State Tracking…arxiv.org
- 6Unite.AI: Ai2 Releases StateBenchunite.ai
QuietHow it’s measured
Hacker News · 4 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna did well on immutable-assignment and reassignment-by-value problems. On the longest problems that require assignment by reference, such as swaps, all four scored 13% or lower. Ai2 ran these evaluations itself, and no independent rerun has been reported. Pretrained open 7B models were near chance on the hardest level. [1] [5]
The benchmark sorts state tracking into four levels of increasing algebraic difficulty, running from immutable assignment to updates by reference. Each hard sequence is paired with three easier versions that keep the same variables, values and positions, so only the type of update changes. An outside replay confirmed that these matched traces agree. [1] [5] [7]
When small 275M-parameter models were trained directly on sequences of up to 256 assignments, Gated DeltaNet and hybrid models stayed near perfect on the easier levels across tens of thousands of assignments. On the hardest level, one variant held up to about 10 times its training length. Transformers trained the same way never learned the hardest level. This extends earlier theory about the limits of state tracking in transformers and state-space models. [1] [5] [2] [3]
accuracy collapsed on dependent updates such as swaps as inputs grew longer, and results shifted sharply with prompt format. Implication: code traces, ledger logs and multi-step agent state may be safer kept in external structured state or tools.
Executive brief
Ai2's new StateBench benchmark finds that four frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna) fall to 13% accuracy or lower on its longest, hardest tasks. Those tasks require following chains of dependent updates, such as swaps or one variable copying another, across a long document (Ai2 blog). The same models handled simple lookups and "latest value" updates well. Small recurrent and hybrid models (Gated DeltaNet) trained directly on the tasks generalized far beyond their training lengths. Transformers trained the same way never learned the hardest level. This matters for anyone relying on LLMs to follow code, logs or agent state over long inputs.
What changed and event timeline
Negative eigenvalues unlock linear-RNN tracking
Grazzi et al. show that letting DeltaNet and Mamba use negative eigenvalues enables parity and improves state tracking, and pretrain such models at 1.3B parameters.
Theory flags state-tracking limits
Merrill, Petty and Sabharwal argue that transformers and Mamba-style SSMs can't solve permutation-composition state tracking. This is the theoretical basis for StateBench.
Olmo Hybrid released
Ai2 ships a fully open 7B model with three Gated DeltaNet layers for every attention layer. It is one of the open models StateBench later tests.
Tech report posted
Petty, Merrill and Sabharwal post the four-level hierarchy, StateBench and new expressivity proofs to arXiv.
Public launch
Ai2 releases the blog post, dataset and code, and reports that frontier models break down on harder levels as inputs grow longer.
More detail
Capabilities and access
Read the full section
- Release: the benchmark, not a model. Dataset and code are public; Unite.AI reports 68,000 rows and an Apache-2.0 license (Unite.AI).
- Models evaluated:
- Open: Olmo 3 7B (transformer) and Olmo Hybrid 7B.
- Smaller trained models: 275M and 2.7B parameters.
- Frontier: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna (Ai2 blog).
Technical analysis for researchers and developers
- Hierarchy: The four levels map to Krohn–Rhodes decompositions of word problems over finite monoids:
- r-trivial: immutable assignment
- aperiodic: reassignment by value
Read the full section
- Hierarchy: The four levels map to Krohn–Rhodes decompositions of word problems over finite monoids:
- r-trivial: immutable assignment
- aperiodic: reassignment by value
- soluble: self-referential updates (x = x + 5)
- general finite: assignment by reference, such as swaps or S₅
- Matched design: Each general-finite sequence is turned into three easier versions. All four keep the same variables, values and assignment positions, so only the update type changes (Ai2 blog). An independent replay confirmed that the generator's matched traces agree, but measured no model accuracy (DEV Community).
- Prompt format matters: Small hybrid models scored near the ceiling on RULER-style prompts but near chance when the same problems were framed as spreadsheet edits.
- Direct-training protocol: Models trained on up to 256 assignments and were tested on up to 64,000.
Claims and evidence
- On the longest general-finite problems, frontier models scored ≤13%. All four did well on r-trivial and aperiodic problems (Ai2 blog).
- Pretrained open models were near chance on general-finite problems, and their accuracy fell on aperiodic problems as length grew (Ai2 blog).
- In the directly trained 275M models:
Read the full section
- On the longest general-finite problems, frontier models scored ≤13%. All four did well on r-trivial and aperiodic problems (Ai2 blog). These are author-run evaluations; no independent rerun was found.
- Pretrained open models were near chance on general-finite problems, and their accuracy fell on aperiodic problems as length grew (Ai2 blog).
- In the directly trained 275M models:
- Gated DeltaNet and hybrid models were near perfect on the three easier levels out to tens of thousands of assignments.
- On the hardest level, the more flexible Gated DeltaNet variant stayed near perfect to about 10× its training length.
- Transformers never learned general-finite problems.
- The paper proves that transformers can't length-generalize on all aperiodic problems (arXiv).
Context and prior work
- StateBench extends the S₅ "illusion of state" argument from the single hardest case to a graded hierarchy (Merrill et al.).
- It builds on the eigenvalue-range line of work for linear RNNs (Grazzi et al., ICLR 2025).
- It tests Ai2's own architecture, Olmo Hybrid, which replaces 75% of attention layers with Gated DeltaNet (skila.ai).
Limitations, safety and contested findings
- Fixed entity set: StateBench uses a fixed number of variables. Real conversations keep introducing new entities, which it does not cover (Ai2 blog).
- Possible tension with prior work:
- Ai2 says Gated DeltaNet with non-negative eigenvalues can track parity under fixed-precision arithmetic with rounding.
Read the full section
- Fixed entity set: StateBench uses a fixed number of variables. Real conversations keep introducing new entities, which it does not cover (Ai2 blog).
- Possible tension with prior work:
- Ai2 says Gated DeltaNet with non-negative eigenvalues can track parity under fixed-precision arithmetic with rounding.
- Grazzi et al. proved that finite-precision linear RNNs with only positive eigenvalues cannot solve parity (Grazzi et al.).
- The two sets of precision assumptions appear to differ, so readers should check the definitions before treating these as conflicting.
- Narrow evidence base: The direct-training results come from 275M-parameter models on synthetic tasks.
Business and practitioner implications
- Don't trust big context windows for dependent state.
- Use structured state, tools or code interpreters rather than relying on in-context tracking.
- Results shifted sharply between prompt formats, so evaluate in your domain's format.
Read the full section
- Don't trust big context windows for dependent state. Workloads that need chained updates, such as code execution traces, inventory or ledger logs and multi-step agent state, are where frontier models broke down (RuntimeWire).
- Track state outside the model. Use structured state, tools or code interpreters rather than relying on in-context tracking.
- Test in your own format. Results shifted sharply between prompt formats, so evaluate in your domain's format.
- Architecture signal: The results support hybrid and recurrent designs where state tracking matters. The gains depend on training data, not just architecture.