AgentsAutonomy & tool use
NVIDIA details the AI agents it uses to diagnose performance problems on its DGX training clusters and releases experimental code
NVIDIA has described a set of MCP-connected AI agents its site reliability engineers use to trace training-performance problems across compute, network, hardware and scheduling on DGX clusters, and published the framework as experimental open-source code that runs inference on-premises.
From the NVIDIA Generative AI article of November 4, 2026 and cited sources: NVIDIA says it runs six domain agents in production on its DGX H100, B200 and GB300 clusters. [1]
SOURCES[1] How NVIDIA Uses AI Agents to Streamline AI Factory… developer.nvidia.com
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- NVIDIA says it runs six domain agents in production on its DGX H100, B200 and GB300 clusters. [1]
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 4, 2026 article published
- Covered
- Nov 5 edition
- Evidence
- 6 sources across 6 sites
- 1How NVIDIA Uses AI Agents to Streamline AI Factory…developer.nvidia.com
- 2Linked video: How NVIDIA Uses AI Agents to Streamline AI…youtube.com
- 3Nvidia wraps its NemoClaw around OpenClaw for the sake of…theregister.com
- 4NVIDIA/AI-Factory-Operations-Agent (GitHub)github.com
- 5NVIDIA Deploys AI Agents to Optimize DGX SuperPOD…blockchain.news
- 6Operate AI Factories with Agents…build.nvidia.com
QuietHow it’s measured
YouTube video · 29 views · 3 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
NVIDIA says it runs six domain agents in production on its DGX H100, B200 and GB300 clusters. They cover job recovery, hardware, observability, research, Kubernetes and cluster management, communicate over MCP, run on locally served Nemotron models and reach SREs through a single chat interface. [1]
The security design limits what the agents can touch. Shell commands run in a separate Kubernetes sandbox for each session. Cluster inspection goes through a pinned, server-side kubectl tool with read-only credentials, and RBAC denies reading secrets or changing pods. NVIDIA recommends a dedicated control plane so the agents never use training GPUs. [1] [3]
NVIDIA's demo used a deliberately injected fault. The agents found a setting that sent GPU-to-network-card traffic through the CPU, applied the fix on both machines when asked, and bandwidth rose from about 13 to about 50 GB/s. [2]
The code is on GitHub under the Apache 2.0 license but is marked experimental. Early access requires an NGC invitation, and APIs and configuration may change between releases. It works with any OpenAI-compatible model endpoint and integrates with Kubernetes, Prometheus or VictoriaMetrics, Grafana, Slurm and Base Command Manager. [4] [6]
The reviewed sources document a concrete architecture—scoped MCP tools, read-only credentials, per-session sandboxes—that operators can replicate.
Read the full assessment
Whether it diagnoses real incidents reliably is unshown, so teams should test it against their own past failures before granting write access.
Executive brief
NVIDIA says it now runs a team of AI agents on its own DGX H100, B200 and GB300 production clusters. The agents trace training-performance problems across compute, network, hardware and scheduling layers, and SREs work with them through one chat interface (NVIDIA blog). In NVIDIA's demo, the agent found a misconfiguration and fixed it, and bandwidth rose from about 13 to about 50 GB/s (video). The fault was injected on purpose for the demonstration. The framework is released as experimental Apache-2.0 code on GitHub. No reviewed source has independently checked how well it performs.
What changed and event timeline
NemoClaw and OpenShell launched at GTC
NVIDIA wrapped the OpenClaw agent platform with Nemotron models and an open-source sandbox runtime called OpenShell, and presented the package as a security layer for risky autonomous agents ().
NVIDIA describes its internal ops agent
The blog shows six domain agents that talk over MCP, run on-premises and are already used in production on DGX SuperPOD clusters ().
Code released as experimental
The GitHub repo uses the Apache 2.0 license, but early access requires an NGC registry invitation. NVIDIA warns that APIs and configuration may change between releases ().
First outside coverage restates NVIDIA's claims
Blockchain.News repeated the announcement without independent testing ().
Capabilities and access
- Agents: Job Recovery, Hardware, Observability, Research, Kubernetes and Cluster Management. Shared MCP tools cover web search, code execution and Grafana (NVIDIA blog).
- Models: the blog says "locally served Nemotron open models" but names no version.
- Integrations: Kubernetes, Prometheus or VictoriaMetrics, Grafana, Slurm and Base Command Manager (GitHub).
Read the full section
- Agents: Job Recovery, Hardware, Observability, Research, Kubernetes and Cluster Management. Shared MCP tools cover web search, code execution and Grafana (NVIDIA blog).
- Models: the blog says "locally served Nemotron open models" but names no version. The repo accepts any OpenAI-compatible endpoint, or a vLLM server managed by the Helm chart (GitHub).
- Integrations: Kubernetes, Prometheus or VictoriaMetrics, Grafana, Slurm and Base Command Manager (GitHub).
- Access: an experimental build (developer example). NVIDIA customers can ask their account manager for access to NVIDIA's own internal agents.
Technical analysis for researchers and developers
- Deployment: it runs as a Kubernetes control-plane app installed with Helm.
- Security: shell commands run in a separate Kubernetes sandbox for each session.
- Recommended setup: a dedicated GPU control plane kept separate from production, so the agents never use training GPUs.
Read the full section
- Deployment: it runs as a Kubernetes control-plane app installed with Helm. The UI, OpenClaw gateway, MCP servers, inference endpoint and OpenShell runtime each run as separate services (NVIDIA blog).
- Security: shell commands run in a separate Kubernetes sandbox for each session. Cluster inspection goes through a server-side
run_kubectltool that uses a pinned binary and read-only credentials, so OpenShell never holds Kubernetes credentials. Kubernetes permissions (RBAC) allow reading pods but deny reading secrets and creating or updating pods. - Recommended setup: a dedicated GPU control plane kept separate from production, so the agents never use training GPUs.
- Evaluation: NVIDIA says it tracks root-cause accuracy and how often SREs agree with the agent, but it has published no numbers, dataset or method. The repo lists no benchmarks (GitHub).
Claims and evidence
- Production use on DGX H100, B200 and GB300 clusters: reported only by NVIDIA (NVIDIA blog).
- Demo results: the baseline was about 13 GB/s (video). This was a single injected fault, not a measured result across real incidents.
- "Has in the past taken hours": no baseline data supports this (video).
Read the full section
- Production use on DGX H100, B200 and GB300 clusters: reported only by NVIDIA (NVIDIA blog).
- Demo results: the baseline was about 13 GB/s (video). The root cause was a setting that sent traffic through the CPU between the GPU and the network card (video). After the fix, bandwidth reached about 50 GB/s, "more than 3x" (video), and stayed there a minute later (video). This was a single injected fault, not a measured result across real incidents.
- "Has in the past taken hours": no baseline data supports this (video).
- Corroboration: no independent evaluation or reporting was found.
Context and prior work
- MCP is Anthropic's open protocol from November 2024 for connecting agents to tools (Blockchain.News).
- NemoClaw and OpenShell came out of NVIDIA's March 2026 push to make OpenClaw-style agents safe enough for businesses.
- The Research agent draws on NVIDIA Mission Control manuals and DGX SuperPOD reference architectures (NVIDIA blog).
Read the full section
- MCP is Anthropic's open protocol from November 2024 for connecting agents to tools (Blockchain.News).
- NemoClaw and OpenShell came out of NVIDIA's March 2026 push to make OpenClaw-style agents safe enough for businesses. That push followed security problems with OpenClaw earlier in the year (The Register).
- The Research agent draws on NVIDIA Mission Control manuals and DGX SuperPOD reference architectures (NVIDIA blog).
Limitations, safety and contested findings
- Read-only versus the demo: NVIDIA's main lesson is to start read-only. The reviewed sources don't explain how that write access is granted or limited.
- Overstated framing: the video says AI factories can now "recursively self-improve" (video). The evidence is one supervised fix.
- Agent risks: NVIDIA's Kari Briski has said such agents can access sensitive data, misuse tools or escalate their own privileges (The Register).
Read the full section
- Read-only versus the demo: NVIDIA's main lesson is to start read-only. In the demo, though, the operator asks the agent to apply the fix, and it changes settings on both machines (video). The reviewed sources don't explain how that write access is granted or limited.
- Overstated framing: the video says AI factories can now "recursively self-improve" (video). The evidence is one supervised fix.
- Agent risks: NVIDIA's Kari Briski has said such agents can access sensitive data, misuse tools or escalate their own privileges (The Register).
- Early-stage code: the repo had 32 stars and access is invitation-only (GitHub).
Business and practitioner implications
- Operators: this is a reference pattern you can copy: narrowly scoped MCP tools, read-only RBAC, a sandbox per session and on-premises inference.
- Before you trust it: build your own root-cause accuracy tests against your past incidents. NVIDIA has published none.
- Strategy: The Register describes NemoClaw as part of NVIDIA's "picks-and-shovels" play to drive compute demand (The Register).
Read the full section
- Operators: this is a reference pattern you can copy: narrowly scoped MCP tools, read-only RBAC, a sandbox per session and on-premises inference. For data-sovereign or multi-tenant sites, tenant SREs see only their own namespaces (NVIDIA blog).
- Before you trust it: build your own root-cause accuracy tests against your past incidents. NVIDIA has published none.
- Strategy: The Register describes NemoClaw as part of NVIDIA's "picks-and-shovels" play to drive compute demand (The Register). Lock-in risk depends on how deeply you adopt Base Command Manager and Mission Control.
The source trail.
Sources (6)
How NVIDIA Uses AI Agents to Streamline AI Factory Operations (NVIDIA Technical Blog)
developer.nvidia.comLinked video: How NVIDIA Uses AI Agents to Streamline AI Factory Operations
linked video manual captions
www.youtube.com