ResearchPapers, evidence & method
Goodfire's Eric Bigelow says LLM answers often hinge on single sampled 'forking' tokens, and argues chain-of-thought is becoming harder to monitor
Eric Bigelow of Goodfire argues that an LLM's final answer is often settled when individual tokens are sampled, not planned in advance, and that as chain-of-thought becomes harder to read, understanding agent behavior will increasingly depend on scalable interpretability tools.
QuietHow it’s measured
YouTube video · 177 views · 2 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
AT Pulse research deskPublished · 5 min read · AI-assisted analysisResampling a model at every position in its output shows that one sampled token can abruptly change the distribution of final answers. These forks sometimes fall on punctuation or spaces. Bigelow concludes that the decision often happens at sampling time, as the model builds on whatever it has already generated. [3] [1]
The follow-up paper, Forking Fast, found that with enough resamples, uncertainty curves are mostly smooth, with sharp jumps only at a few points. Much of the apparent token-level noise comes from sampling itself. Its change-point smoothing method cuts resampling budgets by roughly 8×, but it was tested only on 8B models using tinyMMLU. [2] [9]
Goodfire's Reasoning Theater work used probes to detect when a reasoning model already knows its answer but keeps generating reasoning. Goodfire reports that stopping at 95% probe confidence saved up to 67.7% of tokens on MMLU while keeping 96.7% of accuracy. No independent replication has been found. [8]
Bigelow argues that training rewarded only on outcomes gives models no incentive to keep chain-of-thought readable, and that optimizing it to be monitorable can make it less faithful. He also says interpretability research on frontier-scale open-weights models currently depends on Chinese models such as Kimi K3. [1]
Published experiments show that a single output can hide real uncertainty, which resampling or internal probes can reveal.
Read the full assessment
The broader implication, that chain-of-thought monitoring alone may not be enough to oversee agents, is still Bigelow's argument rather than a demonstrated finding.
Brief update: I've finished the research and am writing the dossier now.
Executive brief
Eric Bigelow of Goodfire argues that an LLM's "decision" often happens at sampling time, not inside the model. A single sampled token, sometimes just punctuation, can flip a model from unsure to certain about its final answer (episode, 00:10:41). His August 2026 paper complicates that picture. With enough resamples, uncertainty curves turn out to be stable, and much of the apparent noise comes from sampling itself rather than from the model reacting to each token (Forking Fast). For anyone relying on chain-of-thought (CoT) monitoring, his point is that reasoning traces are getting harder to read. Understanding why agents act as they do may therefore depend on interpretability tools that work at production scale.
What changed and event timeline
Thought Anchors extends the idea to sentences
Macar, Nanda, Conmy and colleagues resampled reasoning models sentence by sentence. Planning and uncertainty-management sentences turned out to matter most ().
Reasoning Theater
Goodfire used probes to flag "performative" CoT, where the model already knows its answer but keeps reasoning. Probe-based early exit cut tokens substantially ().
Podcast episode
Bigelow tells Nathan Labenz that CoT reliability is declining. He also says frontier open-weights interpretability currently depends on Chinese models ().
Capabilities and access
- Forking Paths: GPT-3.5 via API on seven tasks (CoinFlip, LastLetter, AQuA, GSM8k, MMLU, HotpotQA, StoryCloze) (arXiv).
- Forking Fast: Llama-3-8B-Instruct and DeepSeek-R1-Distill-Llama-8B on tinyMMLU (n=100), using 1.77B collected tokens (arXiv HTML).
- Reasoning Theater: DeepSeek-R1-0528-671B, GPT-OSS-120B and R1 distills, evaluated on MMLU-Redux and GPQA-Diamond (Goodfire).
Read the full section
- Forking Paths: GPT-3.5 via API on seven tasks (CoinFlip, LastLetter, AQuA, GSM8k, MMLU, HotpotQA, StoryCloze) (arXiv).
- Forking Fast: Llama-3-8B-Instruct and DeepSeek-R1-Distill-Llama-8B on tinyMMLU (n=100), using 1.77B collected tokens (arXiv HTML).
- Reasoning Theater: DeepSeek-R1-0528-671B, GPT-OSS-120B and R1 distills, evaluated on MMLU-Redux and GPQA-Diamond (Goodfire).
- Episode claims: Bigelow calls 7–8B parameters the interpretability "sweet spot" and says Goodfire runs infrastructure on Kimi K3 (00:46:41). The reviewed papers do not document either claim.
Technical analysis for researchers and developers
- Forking Paths method: 30 samples per token position on roughly 30 examples per task, about $500 in total.
- Forking Fast method: PELT change-point detection with an exact multinomial cost, then Gaussian-kernel Dirichlet pooling within segments, with hyperparameters tuned by cross-validation.
- Reading the internals: Bigelow says decoding full outcome distributions from hidden states has been hard.
Read the full section
- Forking Paths method: 30 samples per token position on roughly 30 examples per task, about $500 in total. Bayesian change-point detection and survival analysis locate the forks. Between 27% and 63% of sequences had detectable change points (arXiv).
- Forking Fast method: PELT change-point detection with an exact multinomial cost, then Gaussian-kernel Dirichlet pooling within segments, with hyperparameters tuned by cross-validation. Sampling noise falls at the rate 1/√S expected from sampling theory. Error versus the S=200 reference (total variation distance) runs from 0.0265 at S=5 to 0.0056 at S=100 (arXiv HTML).
- Reading the internals: Bigelow says decoding full outcome distributions from hidden states has been hard. In his view, models may represent only a few steps ahead (01:12:41).
Claims and evidence
- Punctuation and spaces can trigger forks (Forking Paths).
- Smoother dynamics in reasoning models. He presents this as his own impression, not a published result.
- Exiting at 95% probe confidence saved up to 67.7% of tokens on MMLU while keeping 96.7% of accuracy (Goodfire).
Read the full section
- Forks at odd tokens. Punctuation and spaces can trigger forks (Forking Paths).
- Smoother dynamics in reasoning models. Bigelow says reasoning models show smoother uncertainty curves and do something like a linearized search: they list many options, then pick one (00:10:41, 00:42:48). He presents this as his own impression, not a published result.
- Early exit saves tokens. Exiting at 95% probe confidence saved up to 67.7% of tokens on MMLU while keeping 96.7% of accuracy (Goodfire). Goodfire reported this itself; no independent replication was found.
- Unsourced episode claims. The episode also mentions a "stolen chain of thought" paper, Astra's unusual messages to its subagents, and a "Hugging Face incident" (01:31:49, 01:53:52). None of these is corroborated in the reviewed sources.
Context and prior work
Read the full section
- Psychology framing. Bigelow's background is Harvard psychology under Tomer Ullman. His profile still lists him as a PhD student and Goodfire research fellow (alphaXiv), while the episode says he has finished the PhD.
- Grokking-style phase transitions. He links sharp per-example jumps to the observation that smooth aggregate learning curves hide stepwise changes on individual tasks (00:23:57).
- Same method, different units. Thought Anchors applies counterfactual resampling to sentences rather than tokens (LessWrong).
Limitations, safety and contested findings
- Forking Paths used one older API model and about 30 examples per task. Forking Fast covers only 8B models on tinyMMLU.
- Abrupt forks may be overstated. Forking Fast attributes much token-level noise to sampling.
- Bigelow argues that outcome-only RL gives no reason to keep CoT human-readable, and that optimizing CoT to be monitorable can make it less faithful (01:32:52).
Read the full section
- Narrow coverage. Forking Paths used one older API model and about 30 examples per task. Forking Fast covers only 8B models on tinyMMLU. Its credible intervals cover the true value 48–64% of the time against a 90% target, and it works less well above a 0.15 forking threshold (arXiv HTML).
- Abrupt forks may be overstated. Forking Fast attributes much token-level noise to sampling. That cuts against reading every jump in small-sample data as a real fork.
- Weak CoT monitors. Bigelow argues that outcome-only RL gives no reason to keep CoT human-readable, and that optimizing CoT to be monitorable can make it less faithful (01:32:52).
Business and practitioner implications
- Uncertainty from one sample is unreliable.
- Possible cost savings. If Reasoning Theater's probe-gated early exit generalizes, it could cut reasoning-token spend.
- Bigelow says frontier open-weights interpretability currently relies on Chinese models, and restrictions on open-weights models would slow safety research (00:54:34).
Read the full section
- Uncertainty from one sample is unreliable. Most sequences in Forking Paths showed low survival rates (arXiv), so answers that matter deserve resampling or probe-based confidence estimates.
- Possible cost savings. If Reasoning Theater's probe-gated early exit generalizes, it could cut reasoning-token spend.
- Dependence on open weights. Bigelow says frontier open-weights interpretability currently relies on Chinese models, and restrictions on open-weights models would slow safety research (00:54:34).
- Harnesses over fine-tuning. He bets that context, harnesses and memory systems will beat per-user fine-tuning, which favors investment in foundation models.