RoboticsMachines in the physical world
SenseNova-led study finds GPT-6-Astra scores high in robot simulators but fails all real-world plug insertions
A joint study tested OpenAI's GPT-6-Astra on 12 benchmarks and 8 robot platforms. It was the top Hugging Face paper on 19 October. The model scored near the ceiling in simulation, but it failed every real plug insertion, ran slowly and used millions of tokens per long task.
According to arXiv 2610.22471 (Oct 2026) [5]: SenseNova-led study finds GPT-6-Astra scores high in robot simulators but fails all real-world plug insertions.
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- huggingface.co article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2610.22471 (Oct 2026) [5] date from arXiv ID
- When
- Original publication date unverified
- Covered
- Oct 20 edition
- Evidence
- 11 sources across 6 sites
Reach: not measured yet
Simulation results did not carry over to the physical world. GPT-6-Astra reached 81.9% on RoboDojo and 87.3% on DexJoCo. On real robots it succeeded in 17 of 18 rigid pick-and-place trials, 2 of 4 cloth folds, 9 of 23 long navigation runs and none of 11 plug insertions. HexaShell, another paper from the same day, also found that tight insertion tasks still mostly fail. [2] [8]
Speed and cost limit the model to planning rather than real-time control. About 90% of humanoid trial time went to inference, so plans could be outdated before they ran, and long episodes used roughly 9M tokens. The model also could not check whether a physical task was actually done, recover from spills or drops, or avoid undoing earlier progress. Earlier tests by PhyMotion likewise judged it too slow for millisecond-level control. [2] [4]
BizBench, also near the top of the day's list, found that every frontier agent it tested broke business rules partway through tasks. Some broke a rule and fixed it before the end, which a check of the final state alone would miss. Claude Opus 5 had the best strict success rate at 64.0%, ahead of GPT-6 Sol at 61.3%. Only 40% of GPT-6 Sol's tasks succeeded on all three attempts. [1] [7]
high simulation scores did not predict success on contact-heavy real tasks, and agents broke rules during runs.
Read the full assessment
Implication: use frontier LLMs as robot planners, not controllers, and score agents on their full execution history, not just the final result.
Researched the Oct 19 Hugging Face Daily Papers page and its top papers; drafting the dossier now.
Executive brief
The most-upvoted paper on Hugging Face Daily Papers for 19 October 2026 (408 upvotes) asks whether OpenAI's general-purpose GPT-6-Astra has "achieved embodied intelligence." Its authors, SenseNova and a joint team of more than 60 researchers, say no. They find the model nearly saturates simulation suites such as LIBERO, yet it failed all 11 real-world plug insertions and burned about 9M tokens per long episode (HF paper page). The day's other leading papers make a similar point. BizBench shows frontier agents breaking business rules partway through tasks, and HexaShell finds tight insertion tasks still mostly fail (HF Daily Papers, Oct 19).
What changed and event timeline
GPT-6 Astra launches
OpenAI positioned the model for computer use, software engineering, science and long-horizon agents. Third-party coverage reports it is available through the API, Azure and AWS Bedrock ().
First embodied claims
36Kr reported that PhyMotion Tech, a Tsinghua spinout, got a 22.48% average success rate across 42 robot tasks with GPT-6 Astra. The article says the model is too slow for millisecond-level control ().
SenseNova paper submitted
The paper tests GPT-6-Astra on 12 benchmarks and 8 real robot platforms. A revision followed on 19 October ().
LEV posted
LEV answers multiple-choice questions by repeating transformer layers in loops instead of generating text. The authors released code and checkpoints ().
Agent-evaluation papers top the daily list
The GPT-6-Astra paper led with 408 upvotes, followed by BizBench (335), HexaShell (289) and LEV (275) ().
Capabilities and access
- GPT-6 Astra (API ID
gpt-6-astra): reported 1.05M-token context window and pricing of $10/$50 per million input/output tokens (Pillitteri). - SenseNova study: names no code or dataset release, only a project homepage (HF).
- HexaShell: tools are open-sourced on GitHub.
Read the full section
- GPT-6 Astra (API ID
gpt-6-astra): reported 1.05M-token context window and pricing of $10/$50 per million input/output tokens (Pillitteri). - SenseNova study: names no code or dataset release, only a project homepage (HF).
- HexaShell: tools are open-sourced on GitHub.
- LEV: LEV-1.4B, 2.6B and 3B models are on HF.
- BizBench: a frozen release that is not public, because it is built on proprietary payment workflows (HF).
Technical analysis for researchers and developers
- SenseNova: uses the same parameters across all platforms and ablates how a slow reasoning model (System 2) works with a fast controller (System 1).
- HexaShell: an "acting" agent attempts each task, and an "optimizer" agent writes new task-specific tools after failures.
- BizBench: scoring is deterministic and checks the full execution history against three tests: goals, process requirements and protection constraints (HF).
Read the full section
- SenseNova: uses the same parameters across all platforms and ablates how a slow reasoning model (System 2) works with a fast controller (System 1). It also tests in-context demonstrations and 10× execution budgets, which roughly tripled RoboDojo success (HF).
- HexaShell: an "acting" agent attempts each task, and an "optimizer" agent writes new task-specific tools after failures. It took 769 optimizer rounds, with a median of 6.3 minutes each. Locating objects by camera triangulation beat Depth Anything V2 by 8.7 points (HF).
- BizBench: scoring is deterministic and checks the full execution history against three tests: goals, process requirements and protection constraints (HF).
- LEV: LoRA fine-tune (rank 16) on looped backbones (Ouro, Nanbeige). A shared pointer head scores the answer options after every loop (HF).
Claims and evidence
No independent replication of any of these results was found.
Read the full section
- GPT-6-Astra simulation: 81.9% on RoboDojo and 87.3% on DexJoCo (HF).
- GPT-6-Astra real world: 17/18 rigid pick-and-place trials, 2/4 cloth folds, 0/11 plug insertions, 9/23 long-horizon navigation runs (HF).
- Conflicting RoboDojo figures: RoboDojo results differ a lot across groups. PhyMotion reports 22.48% success alone and 48% when paired with π0.5 (36Kr; 36Kr). HexaShell reports 39.7% on 42 tasks (HF). The groups used different budgets, harnesses and metrics, so the numbers are not directly comparable.
- BizBench: Claude Opus 5 had the best strict success at 64.0% and GPT-6 Sol 61.3%. Only 40% of GPT-6 Sol's tasks succeeded on all three attempts. Qwen3.5-9B succeeded in 0 of 150 runs (HF).
- LEV: LEV-2.6B rises from 63.2% to 76.2% as depth goes from 1 to 4 loops. Choosing depth per question saves 15–24% of compute at under 1 point accuracy cost. These results come from the authors' own benchmarks, including a newly created one (HF).
No independent replication of any of these results was found.
Context and prior work
- Earlier embodied tests: GPT-6 Astra's robotics performance was being probed weeks before the SenseNova paper.
- Wider theme that week: daily-paper digests from 13–20 October focused on agent verification and self-improving harnesses, such as Self-HPO and HarnessPro (DEV Community).
- Looped models: separate work from the same day reports a "sandwich" looped design beating Ouro (AI Weekly).
Read the full section
- Earlier embodied tests: GPT-6 Astra's robotics performance was being probed weeks before the SenseNova paper. PhyMotion's "Embodied PhyStack" gives low-level control to specialized systems; in its π0.5 pairing, Astra changed only 14.4% of control steps (36Kr).
- Wider theme that week: daily-paper digests from 13–20 October focused on agent verification and self-improving harnesses, such as Self-HPO and HarnessPro (DEV Community).
- Looped models: separate work from the same day reports a "sandwich" looped design beating Ouro (AI Weekly).
Limitations, safety and contested findings
- SenseNova's own list of gaps: in-context demonstration packages don't transfer across tasks. It can't recover from irreversible mistakes like spills and drops.
- Speed: about 90% of humanoid trial time went to inference, so plans could be outdated before they were executed (HF).
- BizBench scope: only 50 tasks, a single user simulator, and a proprietary environment others cannot reproduce (HF).
Read the full section
- SenseNova's own list of gaps: in-context demonstration packages don't transfer across tasks; the model can't verify the physical state, such as whether a task is really complete; it can't recover from irreversible mistakes like spills and drops; and it undoes earlier progress in long tasks. Real robots also can't use extended budgets because of safety and real-time limits (HF).
- Speed: about 90% of humanoid trial time went to inference, so plans could be outdated before they were executed (HF).
- BizBench scope: only 50 tasks, a single user simulator, and a proprietary environment others cannot reproduce (HF).
- Upvotes: upvotes measure attention, not validity.
Business and practitioner implications
- Score the whole run: BizBench found that every model broke rules partway through tasks.
- Budget for retries: models made 1.71× the reference number of tool calls, and success varied across attempts (HF).
- Robotics: use frontier LLMs for planning and correction, not real-time control.
Read the full section
- Score the whole run: BizBench found that every model broke rules partway through tasks. Some DeepSeek runs dipped below cash reserves and restored the balance later, so end-state checks alone would pass them (HF).
- Budget for retries: models made 1.71× the reference number of tool calls, and success varied across attempts (HF).
- Robotics: use frontier LLMs for planning and correction, not real-time control. A long episode can take hours and millions of tokens, and contact-heavy tasks still mostly fail (HF; HF).
- Latency-bound classification: LEV-style adaptive depth is worth testing (HF).