AgentsAutonomy & tool use
Siemens and NVIDIA report that rewriting tools and skills, with the model unchanged, raised a chip-characterization agent's benchmark score from about 55% to 95%
Siemens rewrote the skill files and MCP tool descriptions of a chip-characterization agent, leaving Claude Opus 4.8 unchanged. Evaluated with NVIDIA NeMo Gym, its mean benchmark score rose from about 55% to 95%, while turns and output tokens fell by roughly two-thirds.
From the NVIDIA Generative AI article of October 14, 2026 and cited sources: The model and harness stayed fixed. [1]
SOURCES[1] Optimizing Agent Reliability and Tool Calling Efficiency… developer.nvidia.com
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- The model and harness stayed fixed. [1]
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 14, 2026 article published
- Covered
- Oct 15 edition
- Evidence
- 6 sources across 4 sites
- 1Optimizing Agent Reliability and Tool Calling Efficiency…developer.nvidia.com
- 2Siemens press release: Fuse EDA AI Agent launchnews.siemens.com
- 3Siemens press release: Siemens–NVIDIA at DAC 2026news.siemens.com
- 4eeNews Europe: Siemens advances self-verifying AI for EDAeenewseurope.com
- 5NVIDIA NeMo Gym documentation (v0.2.1)docs.nvidia.com
- 6NVIDIA NeMo Gym release notes (v0.3.0)docs.nvidia.com
QuietHow it’s measured
YouTube video · 262 views · 2 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
The model and harness stayed fixed. The team merged overlapping file and shell tools, added explicit JSON schemas for tool inputs and made the skill text clearer. On a Liberty-file characterization benchmark, the mean score rose from about 55% to 95%. Agent turns fell from 155 to 53, and output tokens fell from about 81K to 29K. [1]
The score depends only on the final library. Four pass/fail gates (coverage, validity, relative validation and integrity) are multiplied by the share of values within tolerance, so any failed gate gives zero. The integrity gate catches outputs that reuse or reformat existing values instead of running the characterization. This matters because the errors here produce plausible-looking results, which step-by-step checks missed. [1]
on one development benchmark, tool schemas, merged tools and clearer skill text improved both accuracy and token use with the model fixed.
Read the full assessment
Implication: teams deploying agents may get more from tuning interfaces and grading final outputs before switching models.
Executive brief
Siemens EDA reports that changing only an agent's skill files and MCP tool descriptions, with the model left as is, raised its mean score on a Liberty-file characterization benchmark from about 55% to about 95%. Over the same work, agent turns fell from 155 to 53 and output tokens from about 81K to 29K. The model was Claude Opus 4.8 running in Claude Code (NVIDIA Technical Blog). NVIDIA NeMo Gym provided the evaluation harness. The scoring check only gives credit for a correct, complete final library, not for tool calls that succeed along the way. No independent group has reproduced the results.
What changed and event timeline
Fuse EDA AI Agent launches
Siemens introduced an autonomous agent for EDA (electronic design automation) at NVIDIA GTC. It supports MCP and Agent Skills, runs on NVIDIA's Agent Toolkit and Nemotron models, and Samsung endorsed it ().
DAC announcement names NeMo Gym
Siemens said agents optimized with NeMo Gym improve quality and token efficiency. It claimed more than 10x faster characterization turnaround and 5–10x lower token costs, with rollout planned through "forthcoming releases" ().
Trade press repeats the figures
eeNews Europe repeated the 10x and 5–10x numbers and listed NeMo Gym, Nemotron, OpenShell and CUDA-X as the technical stack. It added no independent testing ().
Methodology published
An NVIDIA and Siemens blog post describes the verifier, the benchmark setup, and the before/after results for the characterization agent ().
Capabilities and access
- Agent: the Solido Characterization Suite agent, run through Siemens Fuse EDA AI Agent.
- Workflow: tool setup, SPICE simulation with Solido AFS, distributed compute, Liberty (.lib) generation, debugging and validation.
- Model and harness: fixed at Claude Opus 4.8 in Claude Code.
Read the full section
- Agent: the Solido Characterization Suite agent, run through Siemens Fuse EDA AI Agent. It exposes MCP servers and skills to Claude Code, Codex and Cursor (NVIDIA blog).
- Workflow: tool setup, SPICE simulation with Solido AFS, distributed compute, Liberty (.lib) generation, debugging and validation.
- Model and harness: fixed at Claude Opus 4.8 in Claude Code.
- NeMo Gym: an open-source library for building RL environments. It uses the OpenAI Responses API format, sets up and tears down resources for each rollout, and scales to parallel rollouts (NeMo Gym docs). Version 0.3.0 added Claude Code integration (release notes).
- Availability of the Siemens features: "forthcoming releases" (Siemens).
Technical analysis for researchers and developers
- Reward: four pass/fail gates (coverage, validity, relative validation, integrity) multiplied by the fraction of output values within tolerance.
- Setup:
- Benchmark built on the open GF180MCU PDK (process design kit): 229 cells across 36 PVT (process, voltage, temperature) corners.
Read the full section
- Reward: four pass/fail gates (coverage, validity, relative validation, integrity) multiplied by the fraction of output values within tolerance. If any gate fails, the reward is zero. The integrity gate catches outputs where the agent reused, truncated or reformatted existing values instead of running the characterization (NVIDIA blog).
- Setup:
- Benchmark built on the open GF180MCU PDK (process design kit): 229 cells across 36 PVT (process, voltage, temperature) corners.
- Each run used only 2 cells × 36 corners.
- About 25 rollouts in total, each taking 1.5–2 hours.
- What actually changed: overlapping file and shell tools were merged or simplified, tool inputs got explicit JSON schemas, and skill text was clarified. There was no model training. A coding agent drove the loop with a human reviewing each step.
- Reproducibility: the PDK is public. The Solido tools, reference artifacts and verifier code are not.
Claims and evidence
- Score rose from ~55% to ~95%; turns fell 66%; output tokens fell ~64%. These are from the NVIDIA/Siemens authors on a development benchmark (NVIDIA blog).
- More than 10x faster turnaround and 5–10x lower token cost.
- No independent corroboration was found for either set of numbers.
Read the full section
- Score rose from ~55% to ~95%; turns fell 66%; output tokens fell ~64%. These are from the NVIDIA/Siemens authors on a development benchmark (NVIDIA blog). The ~25 rollouts are not broken down into before and after, so variance cannot be judged.
- More than 10x faster turnaround and 5–10x lower token cost. These come only from Siemens marketing (Siemens) and the trade press that repeated them (eeNews). The blog's measured token cut is about 2.8x (81K to 29K), so the two sets of figures measure different things and do not line up.
- No independent corroboration was found for either set of numbers.
Context and prior work
- NeMo Gym was built to supply RL training environments for Nemotron models. Version 0.3.0 shipped more than 70 environments (release notes).
- In this case it is used for evaluation-driven prompt and tool engineering, not RL training.
- The work continues the Siemens–NVIDIA push toward "self-verifying" agents that check their decisions against established, physics-based EDA tools (eeNews; Siemens).
Limitations, safety and contested findings
- Small scope: a two-cell subset of one mature open PDK, about 25 rollouts, one model (NVIDIA blog).
- No held-out evaluation: the same tasks were reused for every round of tuning, so the gains may partly reflect fitting to that benchmark.
- Silent failures are the main risk. Checking only intermediate steps missed these.
Read the full section
- Small scope: a two-cell subset of one mature open PDK, about 25 rollouts, one model (NVIDIA blog).
- No held-out evaluation: the same tasks were reused for every round of tuning, so the gains may partly reflect fitting to that benchmark.
- Silent failures are the main risk. The authors note that errors tend to produce plausible-looking outputs rather than clean failures, such as wrong slew thresholds or missing corners. Checking only intermediate steps missed these.
- Advanced nodes untested: the published method does not cover the advanced-node libraries that Siemens' marketing claims target (Siemens).
Business and practitioner implications
- Tool descriptions, JSON schemas and skill text can matter as much as model choice. Here, changing them cut both errors and token spend.
- Build verifiers that check the end product and can detect shortcuts. The post argues this method transfers to CAE and other engineering domains.
- Budget for cost. Rollouts lasting 1.5–2 hours make evaluation compute-heavy, so plan for it.
Read the full section
- Tune the interface before the model. Tool descriptions, JSON schemas and skill text can matter as much as model choice. Here, changing them cut both errors and token spend.
- Grade the final artifact. Build verifiers that check the end product and can detect shortcuts. The post argues this method transfers to CAE and other engineering domains.
- Budget for cost. Rollouts lasting 1.5–2 hours make evaluation compute-heavy, so plan for it.
- For buyers: treat the 10x and 5–10x claims as unverified until Siemens publishes per-workload benchmarks.
The source trail.
Sources (6)
Optimizing Agent Reliability and Tool Calling Efficiency for Chip Design Using NVIDIA NeMo Gym
Article text retrieved; extracted text may omit tables or interactive elements. A linked video was found but its transcript could not be retrieved (a0UtM7bPKts: captions unavailable (TranscriptsDisabled); a0UtM7bPKts: audio transcription failed (Audio transcription returned HTTP 429)); do not claim to have watched it.
developer.nvidia.com