RoboticsMachines in the physical world
Ai2 releases ToyBlocks, an open simulated block world whose simulation-trained robot stack ran on a real Franka arm
Ai2's ToyBlocks, the first of its FoundationWorlds simulations, trains a planner and controller only on generated tabletop-block data. It ships with open data, models and code. Its robot stack transferred to a real Franka arm, and its tests showed popular action models generalize poorly.
From the Hugging Face article of November 6, 2026 and cited sources: The robot stack was trained only in simulation, with no real-world training. [1]
SOURCES[1] ToyBlocks: A controlled world for robot learning and… huggingface.co
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- The robot stack was trained only in simulation, with no real-world training. [1]
- Original source
- Hugging Face article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2509.01819 (Sep 2025) [2] date from arXiv ID
- When
- November 6, 2026 article published
- Covered
- Nov 7 edition
- Evidence
- 8 sources across 6 sites
QuietHow it’s measured
Hacker News · 2 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
The robot stack was trained only in simulation, with no real-world training. On a physical Franka arm, individual skills succeeded 81.1% of the time, against 80.0% in simulation. Full three-block spelling tasks succeeded in 15 of 30 real trials (50%), against 55.5% in simulation. [1]
ToyBlocks is designed so robot failures can be traced to a cause. Training data is generated to order, so specific cases such as four-block stacks or one region of the table can be held out for testing. Oracle planners and controllers separate planning failures from execution failures. Results are scored on speed, contact forces and task progress as well as success. The release includes 85,336 demonstrations, a Molmo 2 planner and a ManiFlow controller. [1] [5] [8]
On held-out splits, widely used action models (ManiFlow, π0.5 and MolmoAct 2) generalized poorly, though LoRA and other techniques improved results. This fits earlier LIBERO-Plus findings that vision-language-action models rely on memorized object positions. [1] [4]
A Claude Opus 5.5 agent solved all 30 classical stacking tasks but still misread look-alike letters and letter orientation, so perception, not planning, held it back. Separately, adding 100 Wan 2.2 world-model rollouts raised policy success in an unseen table region from 19% to 78%. [1]
training only in simulation matched simulated skill success on real hardware, and held-out splits exposed weak generalization in action models.
Read the full assessment
Implication: teams could test data mixtures and claims about pretrained models cheaply before paying for teleoperation, though block-world findings may not carry over to open-ended tasks.
Executive brief
Ai2 trained a planner and a robot controller only on simulated tabletop-block data, then ran them on a physical Franka arm with no real-world training. The individual skills worked about as well on the real arm as in simulation: 81.1% vs 80.0% success. Full three-block spelling tasks succeeded 50% of the time on the real arm (15/30) and 55.5% in simulation (Ai2). ToyBlocks is the first of Ai2's "FoundationWorlds": small, controlled simulations meant to make robot failures traceable. It comes with open data, models and code. Ai2's own tests found that widely used action models generalize poorly within this deliberately simple world.
What changed and event timeline
LIBERO-Plus documents VLA brittleness
This study changed one factor at a time across seven perturbations. VLA models failed when target objects moved and dropped sharply under lighting changes ().
Trade press summarizes
Unite.AI and RuntimeWire covered the release. Both relied on Ai2's materials and included no outside expert commentary (;).
ToyBlocks released
Ai2 announced FoundationWorlds and released ToyBlocks with simulation assets, task generators, 85,336 demonstrations and two trained models (;).
Capabilities and access
- Release contents: simulation assets, procedural task and demonstration generators, training code, evaluation code and validation sets (Ai2).
- Models:
allenai/Molmo2-4B-tb(high-level planner) andallenai/maniflow-tb(low-level controller) (Ai2). - Licensing: ManiFlow-tb weights and code are Apache-2.0 and research-only (model card). Unite.AI reports the data is under ODC-BY in LeRobot format (Unite.AI).
Read the full section
- Release contents: simulation assets, procedural task and demonstration generators, training code, evaluation code and validation sets (Ai2).
- Models:
allenai/Molmo2-4B-tb(high-level planner) andallenai/maniflow-tb(low-level controller) (Ai2). - Licensing: ManiFlow-tb weights and code are Apache-2.0 and research-only (model card). Unite.AI reports the data is under ODC-BY in LeRobot format (Unite.AI).
- Data: about 1,600 hours and 144M frames. A larger second release is planned "in a few weeks."
- Web interface: lets Claude or another frontier model drive the robot in natural language.
Technical analysis for researchers and developers
- Data generation: RL and motion planning in Nvidia Isaac Lab learn short skills (push, flip, rotate).
- Hierarchy: Molmo 2 picks a skill and points at the target block. ManiFlow executes it, then pauses to ask for the next instruction.
- Evaluation: metrics go beyond success rate to include execution speed, contact forces and task progress.
Read the full section
- Data generation: RL and motion planning in Nvidia Isaac Lab learn short skills (push, flip, rotate). They use privileged simulator state, such as exact block poses, that a real robot would not have. A task-and-motion planner then chains these skills into labeled demonstrations, running hundreds of plans in parallel per GPU at up to 24× real time (Ai2).
- Hierarchy: Molmo 2 picks a skill and points at the target block; ManiFlow executes it, then pauses to ask for the next instruction. Oracle planners and controllers let researchers tell planning failures apart from execution failures.
- Evaluation: metrics go beyond success rate to include execution speed, contact forces and task progress.
- Sim-to-real steps: system identification (joint friction, armature), randomized physics and visuals, and depth maps estimated from stereo images with FastFoundationStereo.
- Reproducibility: simulation is non-deterministic. On 128 held-out episodes, ManiFlow-tb succeeded 54–59% of the time across four runs, and runs agreed on only 82–97 episodes (model card).
Claims and evidence
All results below are reported by Ai2. No independent replication was found.
Read the full section
All results below are reported by Ai2. No independent replication was found.
- Sim-to-real: skills 80.0% (sim) vs 81.1% (real); full tasks 55.5% (111/200, sim) vs 50.0% (15/30, real). Ai2 says more trials are needed (Ai2).
- VLA generalization: ManiFlow, π0.5 and MolmoAct 2 generalized poorly. LoRA and other techniques improved results.
- Claude Opus 5.5 agent: solved 30/30 classical stacking tasks. In search-and-spell (half the scenes impossible), it got 63% right. Errors included look-alike letters (13%) and wrong letter orientation (15%).
- Agent-written skill descriptions: raised success to 63%, compared with 24% for the default descriptions and 36% for a human expert's.
- World models: Wan 2.2 predicted block motion and outcomes most accurately. Adding 100 Wan 2.2 rollouts raised policy success in an unseen region from 19% to 78%.
Context and prior work
ToyBlocks builds on Ai2's MolmoAct 2, MolmoBot and MolmoSpaces (Ai2). Ai2 says earlier studies of VLA generalization reached conflicting conclusions because real-world datasets offer limited spatial variation. LIBERO-Plus is an example of the brittleness side: it found VLAs rely on memorized object positions (LIBERO-Plus).
Read the full section
ToyBlocks builds on Ai2's MolmoAct 2, MolmoBot and MolmoSpaces (Ai2). Ai2 says earlier studies of VLA generalization reached conflicting conclusions because real-world datasets offer limited spatial variation. LIBERO-Plus is an example of the brittleness side: it found VLAs rely on memorized object positions (LIBERO-Plus). ToyBlocks' contribution is generating training data to order, such as holding out four-block stacks or one region of the table, so these questions can be tested directly.
Limitations, safety and contested findings
- Small real-world sample: 30 episodes, all in the easiest setting, with letters visible and upright (Ai2).
- Safety: the ManiFlow-tb model card says the model has not undergone safety benchmarking.
- Perception errors: frontier-model agents still misread letters and orientations despite strong planning.
Read the full section
- Small real-world sample: 30 episodes, all in the easiest setting, with letters visible and upright (Ai2).
- Safety: the ManiFlow-tb model card says the model has not undergone safety benchmarking. It also needs a 194-second evaluation timeout, about 3× the expert time (model card).
- Perception errors: frontier-model agents still misread letters and orientations despite strong planning.
- Inconsistent model naming: the blog says the agent experiments used Claude Opus 5.5, but one video caption names "Claude Fable 5.1."
- Narrow scope: findings from a block world may not carry over to open-ended domains, which is why Ai2 chose this narrow setting.
Business and practitioner implications
- Cheaper experiments: generating demonstrations automatically removes teleoperation cost, so teams can test data mixtures before collecting real data.
- Testing vendor claims: the held-out generalization splits are a low-cost way to check claims about pretrained VLAs.
- Agent evaluation: the MCP-based tool interface lets teams test LLM agents on physical tasks; perception, not planning, was the bottleneck.
Read the full section
- Cheaper experiments: generating demonstrations automatically removes teleoperation cost, so teams can test data mixtures before collecting real data.
- Testing vendor claims: the held-out generalization splits are a low-cost way to check claims about pretrained VLAs.
- Agent evaluation: the MCP-based tool interface lets teams test LLM agents on physical tasks; perception, not planning, was the bottleneck.
- Synthetic training data: the world-model result suggests synthetic rollouts can fill gaps in coverage, though only Ai2 has validated this so far.
- Benchmarking practice: given the 54–59% run-to-run spread, report multiple runs before comparing methods.
The source trail.
Sources (8)
ToyBlocks: A controlled world for robot learning and evaluation
Article text retrieved; extracted text may omit tables or interactive elements.
huggingface.co