InfrastructureCompute, chips & cloud
NVIDIA publishes Kubernetes recipes for Vera Rubin, reporting 744.71 GB/s NVLink all-reduce across two trays
NVIDIA's open-source AI Cluster Runtime now offers Preview recipes that deploy training and inference stacks for Vera Rubin NVL72 on Kubernetes. A two-tray NVLink all-reduce test reached 744.71 GB/s, and the recipes document driver, networking and memory pitfalls.
From the NVIDIA Generative AI article of November 2, 2026 and cited sources: AICR v1.0.0 is licensed Apache 2.0. [1] [6] [8]
SOURCES[1] Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One developer.nvidia.com[6] NVIDIA/aicr GitHub github.com[8] AICR component catalog docs.nvidia.com
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 2, 2026 article published
- Covered
- Nov 4 edition
- Evidence
- 9 sources across 5 sites
- 1Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day Onedeveloper.nvidia.com
- 2Nvidia CEO confirms Vera Rubin NVL72 in production…tomshardware.com
- 3dra-driver-nvidia-gpu v0.5.0 releasegithub.com
- 4Vera Rubin NVL72 cloud availability (Spheron)spheron.network
- 5vLLM Release 26.10 (NVIDIA)docs.nvidia.com
- 6NVIDIA/aicr GitHubgithub.com
Reach: not measured yet
AICR v1.0.0 is licensed Apache 2.0. It bundles existing NVIDIA cloud-native components, including GPU Operator, the DRA driver and KAI Scheduler, into versioned recipes for the vr200 accelerator, with training, inference and Dynamo variants. The recipes are still marked Preview. The validated stack is: R615.23 open driver, GPU Operator v26.7.0, DRA driver v0.5.0, Kubernetes 1.34.1 or later, and Ubuntu 26.04 on a 64k-page arm64 kernel. [1] [6] [8]
Validation measured 744.71 GB/s NCCL all-reduce bus bandwidth at 16 GiB across two compute trays. The run also passed all 4 deployment checks, 8 conformance checks and 1 performance check. It did not cover a full 72-GPU rack, MIG, RDMA data-path performance over ConnectX-9, or inference throughput. [1]
Moving from GB200 is not a drop-in change. Each Rubin tray exposes eight ConnectX-9 device groups, compared with four ConnectX-7 adapters on GB200, so GPU-to-NIC affinity has to be checked again. A static host IMEX service conflicts with DRA-managed ComputeDomains. NFD's default memory limit gets OOM-killed on multi-terabyte nodes. A Rubin bug also crashed the DRA driver; NVIDIA says a go-nvlib update fixed it about three days later and shipped in DRA v0.5.0. The v0.5.0 release notes do not mention the fix. [1] [3]
NVIDIA's own tests show the stack deploys and passes conformance on Rubin hardware, with no independent reproduction.
Read the full assessment
Implication: teams running their own racks can avoid some rediscovery, but production SLAs and inference benchmarks are premature while the recipes remain Preview.
Executive brief
NVIDIA's first published performance test of Vera Rubin under Kubernetes reached 744.71 GB/s NCCL all-reduce bus bandwidth across two compute trays. The test used NVLink and NVIDIA's open-source AI Cluster Runtime (AICR) v1.0.0 (NVIDIA blog). The Vera Rubin recipes are labeled "Preview." The blog says inference performance on Rubin can't be validated yet, because vLLM's own published images were not built with the CUDA version Rubin kernels require. Early testing also turned up a Rubin bug that crashed the GPU allocation (DRA) driver. NVIDIA says it fixed the bug in about three days.
What changed and event timeline
Vera Rubin NVL72 enters production
Jensen Huang used his CES keynote to say the rack-scale system was in production ().
DRA driver v0.5.0 ships
The release adds support for GPU Operator v26.7.0 and an alpha host-managed IMEX mode for ComputeDomains (). NVIDIA says the release includes the Rubin go-nvlib fix ().
First general cloud availability
According to a third-party roundup, Oracle Cloud made Vera Rubin NVL72 generally available on this date. The same roundup says CoreWeave and Google Cloud were each running one named customer ().
Rubin support in NGC vLLM container
NVIDIA's 26.10 vLLM container is built on CUDA 13.4.3 and adds initial support for Rubin ().
AICR Vera Rubin recipes published
The training, inference and Dynamo recipes come with signed validation evidence ().
Capabilities and access
- Tool: AICR v1.0.0, licensed Apache 2.0. The repo lists the vr200 accelerator as Preview (NVIDIA/aicr).
- Recipe target: service rke2, accelerator vr200, OS Ubuntu, with training, inference and Dynamo inference variants.
- Validated stack: host-installed R615.23 open driver, GPU Operator v26.7.0, DRA driver v0.5.0, Kubernetes ≥1.34.1, Ubuntu 26.04 with a 64k-page arm64 kernel.
Read the full section
- Tool: AICR v1.0.0, licensed Apache 2.0. The repo lists the vr200 accelerator as Preview (NVIDIA/aicr).
- Recipe target: service rke2, accelerator vr200, OS Ubuntu, with training, inference and Dynamo inference variants. The training recipe resolves to 14 components across 5 overlays.
- Validated stack: host-installed R615.23 open driver, GPU Operator v26.7.0, DRA driver v0.5.0, Kubernetes ≥1.34.1, Ubuntu 26.04 with a 64k-page arm64 kernel.
- Not covered: MIG, RDMA data-path performance over ConnectX-9, and inference performance.
- Driver branches: early support targets R610, with official platform support in R620. R615.23 is the version tested, not a confirmed minimum (NVIDIA blog).
Technical analysis for researchers and developers
- Workflow: recipe generation runs offline.
aicr bundlethen renders ordered Helm charts with a single deploy script. - Bundle outputs: Helm, Helmfile, Argo CD, Flux, Terraform and Fleet. Artifacts claim SLSA Build Level 3 provenance and include SBOMs.
- Changes from GB200: each tray exposes eight ConnectX-9 PCI device groups (32 functions), versus four ConnectX-7 adapters on GB200. GPU-to-NIC affinity therefore needs rechecking.
Read the full section
- Workflow: recipe generation runs offline.
aicr bundlethen renders ordered Helm charts with a single deploy script. Next comessnapshot, followed by validation in four phases: readiness, deployment, conformance and performance (AICR docs). - Bundle outputs: Helm, Helmfile, Argo CD, Flux, Terraform and Fleet. Artifacts claim SLSA Build Level 3 provenance and include SBOMs.
- Changes from GB200: each tray exposes eight ConnectX-9 PCI device groups (32 functions), versus four ConnectX-7 adapters on GB200. GPU-to-NIC affinity therefore needs rechecking.
- Platform gotchas:
- A static nvidia-imex service on the host conflicts with DRA-managed ComputeDomains.
- The ComputeDomain CRD shipped with GPU Operator v26.7.0 requires
numNodeson fresh installs. - NFD's default 60Mi memory limit gets OOM-killed on multi-terabyte workers (NVIDIA blog).
Claims and evidence
All performance and timing figures below are reported by NVIDIA. No independent reproduction was found.
Read the full section
All performance and timing figures below are reported by NVIDIA. No independent reproduction was found.
- Bandwidth: 744.71 GB/s busbw at 16 GiB, measured across two trays (NVIDIA blog).
- Validation checks: all 4 deployment, 8 conformance and 1 performance checks passed.
- Timings: a two-node ComputeDomain reached running pods in about 12 seconds, and the full deployment-plus-conformance sequence took about 5 minutes.
- Bring-up speed: cluster handoff to a healthy runtime took hours, and a second cluster reproduced the result the next day.
- Bug fix: go-nvlib moved from 0.11.0 to 0.12.0 about three days after the crash report. The DRA v0.5.0 release notes don't mention Rubin or go-nvlib (release).
- Evidence status: the signed evidence won't match today's Preview recipes by digest. NVIDIA describes it as a historical record, not a current attestation.
Context and prior work
- AICR packages existing NVIDIA cloud-native components into versioned recipes: GPU Operator, DRA driver, KAI Scheduler, NVSentinel and NodeWright (AICR component catalog).
- Vera Rubin NVL72 is the third generation of NVIDIA's MGX rack design after GB200 NVL72.
- Multi-node NVLink through IMEX ComputeDomains was already the pattern on GB200 (DRA docs).
Read the full section
- AICR packages existing NVIDIA cloud-native components into versioned recipes: GPU Operator, DRA driver, KAI Scheduler, NVSentinel and NodeWright (AICR component catalog).
- Vera Rubin NVL72 is the third generation of NVIDIA's MGX rack design after GB200 NVL72. ConnectX-9 SuperNICs and BlueField-4 DPUs replace ConnectX-7 and BlueField-3 (NVIDIA blog).
- Multi-node NVLink through IMEX ComputeDomains was already the pattern on GB200 (DRA docs).
Limitations, safety and contested findings
- Small scale: the performance gate covers two trays, not a full 72-GPU rack. The blog publishes no other workload benchmarks.
- Hardware issue: early validation was blocked by a fabric problem that kept one tray out of the NVLink fabric until it was reseated.
- Host-managed IMEX limits: it is alpha and isolates by domain only. Every workload in a domain gets channel 0 (release).
Read the full section
- Small scale: the performance gate covers two trays, not a full 72-GPU rack. The blog publishes no other workload benchmarks.
- Hardware issue: early validation was blocked by a fabric problem that kept one tray out of the NVLink fabric until it was reseated.
- Host-managed IMEX limits: it is alpha and isolates by domain only. Every workload in a domain gets channel 0 (release).
- Possible conflict on vLLM: the blog says upstream vLLM images through v0.28.0 lack compiled Rubin kernels. NVIDIA's own NGC container, by contrast, ships vLLM 0.31.0 on CUDA 13.4.3 with initial Rubin support (vLLM 26.10). The blog doesn't reconcile the two.
Business and practitioner implications
- Upgrade planning: teams on GB200 face changes to the driver branch, OS, kernel page size, NICs and DOCA (3.3.0 is the first GA release supporting ConnectX-9).
- Inference timing: benchmarking or SLAs on Rubin should wait for runtime images built with CUDA ≥13.4 that have been verified on real hardware.
- Market context: cloud availability is still uneven. AWS, Azure, Lambda and Nebius had no confirmed live instances as of late October (Spheron).
Read the full section
- Upgrade planning: teams on GB200 face changes to the driver branch, OS, kernel page size, NICs and DOCA (3.3.0 is the first GA release supporting ConnectX-9). Recipes reduce rediscovery but come with no production support while they are Preview.
- Inference timing: benchmarking or SLAs on Rubin should wait for runtime images built with CUDA ≥13.4 that have been verified on real hardware.
- Market context: cloud availability is still uneven. AWS, Azure, Lambda and Nebius had no confirmed live instances as of late October (Spheron). In practice, early users of these recipes will mostly be people who operate their own racks.
The source trail.
Sources (9)
Making Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Article text retrieved; extracted text may omit tables or interactive elements.
developer.nvidia.com