AgentsAutonomy & tool use
Realtime-Bench finds the best frontier model completes 44% of timed agent tasks; Google DeepMind previews Genie 4 world model
A new 50-task benchmark finds frontier models complete at most 44% of timed web and game tasks when the environment keeps moving during inference, versus 70% with the clock paused. Import AI 479 also covers Google DeepMind's Genie 4 world model.
From the Import AI article of November 2, 2026 and cited sources: Realtime-Bench tests agents on 50 timed web and game tasks. [3] [4] [1]
SOURCES[3] Realtime-Bench (arXiv HTML) arxiv.org[4] Realtime-Bench (arXiv abstract) arxiv.org[1] Import AI 479 jack-clark.net
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- Import AI article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2610.22155 (Oct 2026) [4] date from arXiv ID
- When
- November 2, 2026 article published
- Covered
- Nov 3 edition
- Evidence
- 10 sources across 9 sites
- 1Import AI 479jack-clark.net
- 2Linked video: Import AI 479: Spacewar; low-latency tasks…youtube.com
- 3Realtime-Bench (arXiv HTML)arxiv.org
- 4Realtime-Bench (arXiv abstract)arxiv.org
- 5The Next Web: Metis Reasoningthenextweb.com
- 6Intelligence Community News: DIA space reportintelligencecommunitynews.com
StrongHow it’s measured
YouTube video · 596.6K views · 11.8K likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
Realtime-Bench tests agents on 50 timed web and game tasks. Under its Wall Clock condition, the environment keeps changing while the model computes. The best model there, GPT-6-Astra, completes 44% of tasks, but only 36% under Pass³, which counts a task only if all three trials succeed. With the environment frozen during inference, Claude Opus 5 and Gemini 3.8 Flash both reach 70% at high reasoning effort. [3] [4] [1]
Under time pressure, more reasoning effort sometimes lowers scores, and streaming models did not clearly beat standard ones. The benchmark scores perceiving changes, choosing how long to think and timing actions as separate skills. A Token Clock condition advances simulated time with each generated token, which separates time-awareness from serving speed. The code and a leaderboard are public. [3] [9] [10]
Genie 4 is in research preview for trusted testers only. It turns photos or prompts into explorable worlds with native audio and accepts plain-language edits while running, with Gemini planning and checking each frame against the plan. According to Google alone, with no independent evaluation, scenes stay consistent for up to two minutes. Resolution and frame rate are not disclosed. [7] [8] [1] [2]
Metis Reasoning, a 12-person London lab co-founded by AlphaGo researcher Thore Graepel, proposes a hybrid symbolic-neural reasoning design. It combines a store of claims with confidence levels, a learned policy and a judge that admits only evidence-backed claims. Its position paper includes no experiments or benchmark figures. [5] [1]
with the clock running, no tested model completed half the timed tasks. Implication: agents for bidding, booking or games likely need human fallback and explicit control over thinking time and action timing. Faster models alone may not fix this.
Executive brief
Realtime-Bench is a new benchmark of 50 timed web and game tasks. When the environment keeps moving while a model thinks, the best frontier model, GPT-6-Astra, finishes 44% of tasks, and only 36% reliably across three repeated runs. With the clock paused, top models reach 70% (arXiv). The gap points to where agents will fail in practice: auctions, ticket sales, games. Jack Clark's Import AI 479 puts this alongside Google DeepMind's Genie 4 world model, Metis Reasoning's position paper on a hybrid symbolic-neural design, and a DIA report on space warfare.
What changed and event timeline
Realtime-Bench posted
Researchers from UT Austin, Ohio State, UIUC, NVIDIA, Northwestern and Steadyworks released a benchmark that scores both what an agent does and when it acts. Version 3 followed on Oct 22 ().
Metis leaves stealth
Thore Graepel, an AlphaGo researcher formerly at DeepMind, co-founded this 12-person London lab. It published "The Age of Reasoning" white paper but no benchmarks, and did not disclose its funding ().
DIA space report
This is the third unclassified edition of Challenges to Security in Space, with information current as of July 2026. It covers China, Russia, Iran and North Korea ().
Genie 4 announced
It launched as a research preview for trusted testers, with native audio, real-time editing and Gemini acting as a planning agent ().
Import AI 479
Clark summarizes all four items and calls Genie 4 "quite janky" but a sign of where entertainment and synthetic data are heading ().
Capabilities and access
- Genie 4: You can turn a photo or a prompt into a world you can explore, then change the weather, camera or characters with plain-language edits while it runs. Access is limited to trusted testers for now.
- Promo video: The video shows a picture becoming an environment with ambient sound (0:31), a text prompt becoming a playable scene "a few seconds later" (1:13), and robots practicing tasks in simulation (1:36).
Read the full section
- Genie 4: You can turn a photo or a prompt into a world you can explore, then change the weather, camera or characters with plain-language edits while it runs. It generates sound natively and runs with a Gemini 3.8 Flash planning loop (Import AI; DeepMind). Access is limited to trusted testers for now. Google AI Ultra subscribers (in Project Genie) and Gemini API / AI Studio early-access partners come next.
- Promo video: The video shows a picture becoming an environment with ambient sound (0:31), a text prompt becoming a playable scene "a few seconds later" (1:13), and robots practicing tasks in simulation (1:36).
Technical analysis for researchers and developers
- Realtime-Bench test conditions: It runs under three clocks (arXiv):
- Wall Clock: the environment advances in real time while the model computes, so inference latency counts.
- Paused Clock: the environment freezes during inference, giving an upper bound.
Read the full section
- Realtime-Bench test conditions: It runs under three clocks (arXiv):
- Wall Clock: the environment advances in real time while the model computes, so inference latency counts.
- Paused Clock: the environment freezes during inference, giving an upper bound.
- Token Clock: simulated time advances per generated token, which measures time-awareness separately from serving speed.
- What it scores: perceiving changes (RT-Perceive), adjusting how long to think (RT-Deliberate), and timing actions (RT-Act). Pass³ counts a task only if it succeeds in all three trials.
- Model groups: It compares standard asynchronous models, streaming models (GPT-Realtime-2, the Gemini Live models) and decision models (OpenAI Decisions, Jev 1.13). A GitHub repo and leaderboard are public.
- Genie 4 design: DeepMind staff call it a "hybrid world model". Gemini writes code and text, Genie renders frames, and Gemini checks each frame against its plan (TechCrunch). Resolution and frame rate are not disclosed.
Claims and evidence
- Wall Clock results: best success rate 44% and best Pass³ 36%, both from GPT-6-Astra (arXiv).
- Paused Clock results: Claude Opus 5 and Gemini 3.8 Flash both reach 70% with high reasoning effort.
- Under time pressure: more reasoning effort sometimes hurts results. Streaming models did not clearly beat standard ones (arXiv).
Read the full section
- Wall Clock results: best success rate 44% and best Pass³ 36%, both from GPT-6-Astra (arXiv). Import AI calls the 36% figure a "pass rate" without saying it is the strict three-trial metric.
- Paused Clock results: Claude Opus 5 and Gemini 3.8 Flash both reach 70% with high reasoning effort. Import AI's mention of "even 80%" does not appear in the reviewed paper text.
- Under time pressure: more reasoning effort sometimes hurts results. Streaming models did not clearly beat standard ones (arXiv).
- Genie 4 duration: TechCrunch reports scenes stay consistent for up to two minutes, double Genie 3. DeepMind's page says "a few minutes" of continuous interaction. These claims come from Google alone; no independent evaluation exists.
- Metis: it says results on frontier reasoning problems are "encouraging" and cheaper, but has published no figures (The Next Web).
Context and prior work
Read the full section
- Genie lineage: Genie 4 builds on Genie 2 and Genie 3 (Wikipedia)). DeepMind positions it as a training environment for agents like SIMA and for robotics (DeepMind).
- Metis: it frames its project as doing for open-ended problems what AlphaGo did for board games. The design combines an explicit store of claims with confidence levels, a learned policy and value function, and a separate judge that admits only claims backed by evidence (Import AI).
- DIA report: China went from 22 launches in 2016 to 93 in 2025 (Import AI). Clark links this to future space-based data centers.
Limitations, safety and contested findings
- Genie 4 (acknowledged by Google): details drift over long sessions, multi-agent interaction is limited, audio is not yet accurate, and in-world text is hard to read.
- Realtime-Bench scope: 50 tasks is a small set, and Wall Clock scores depend on serving infrastructure, which the Token Clock condition is designed to control for.
- Metis: this is a position paper with no experiments, so its central claim that its design reduces hallucinations is untested.
Read the full section
- Genie 4 (acknowledged by Google): details drift over long sessions, multi-agent interaction is limited, audio is not yet accurate, and in-world text is hard to read. All output carries a SynthID watermark (DeepMind; TechCrunch). Clark also notes robotic-sounding voices and dreamlike inconsistency.
- Realtime-Bench scope: 50 tasks is a small set, and Wall Clock scores depend on serving infrastructure, which the Token Clock condition is designed to control for.
- Metis: this is a position paper with no experiments, so its central claim that its design reduces hallucinations is untested.
Business and practitioner implications
- Time-sensitive agents (bidding, booking, trading, games) should not be deployed without human fallback.
- "Low-latency" models do not fix the problem on their own.
- Genie 4 points to new uses in previsualization, storyboarding and synthetic training data.
Read the full section
- Time-sensitive agents (bidding, booking, trading, games) should not be deployed without human fallback. Even the top model completes fewer than half of the timed tasks.
- "Low-latency" models do not fix the problem on their own. Systems need to decide how long to think, track stale state, and decide when to act.
- Genie 4 points to new uses in previsualization, storyboarding and synthetic training data. Production use should wait for API access, published specs and independent tests.
- Satellite-dependent infrastructure faces a more contested space environment, according to the DIA.
The source trail.
Sources (10)
Import AI 479
jack-clark.netLinked video: Import AI 479: Spacewar; low-latency tasks versus LLMs; Genie 4
linked video manual captions
www.youtube.com