InfrastructureCompute, chips & cloud
NVIDIA open-sources Recovery, a service that automatically restarts failed distributed training jobs on Slurm
NVIDIA released Recovery under Apache 2.0, a Slurm-based cluster service that detects crashed, hung or slow training jobs, excludes suspect nodes and restarts work from checkpoint. Its own repository labels the project experimental, and native Kubernetes support remains on the roadmap.
From the NVIDIA Generative AI article of November 2, 2026 and cited sources: Recovery runs a four-stage loop. [1] [2] [5]
SOURCES[1] Keep Distributed Training Running with Automated Workload… developer.nvidia.com[2] dsx-ai-factory/NVIDIA-Recovery (GitHub) github.com[5] NVIDIA Recovery FAQ (NVIDIA docs) docs.nvidia.com
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- November 2, 2026 article published
- Covered
- Nov 3 edition
- Evidence
- 6 sources across 6 sites
- 1Keep Distributed Training Running with Automated Workload…developer.nvidia.com
- 2dsx-ai-factory/NVIDIA-Recovery (GitHub)github.com
- 3Revisiting Reliability in Large-Scale Machine Learning…arxiv.org
- 4HyperAI storyhyper.ai
- 5NVIDIA Recovery FAQ (NVIDIA docs)docs.nvidia.com
- 6NVIDIA Resiliency Extension docsnvidia.github.io
Reach: not measured yet
Recovery runs a four-stage loop. It detects failures from log patterns, missing progress signals and sustained throughput drops, classifies the failure, excludes implicated nodes across restart attempts, and then restarts or stops the job through an Advisor API. Jobs opt in with a single sbatch directive. Deployments start in monitor mode, and the tool neither creates checkpoints nor repairs hardware. [1] [2] [5]
NVIDIA reports three months of monitoring on one of its own clusters running nearly 200 jobs a day: about 8,550 detections, with mean latencies of 69 seconds to detect a problem and 94 seconds to act. The 94-second figure measures time to the automated action, not time until training resumes. [1]
Meta's 11-month study of clusters with 24,000 A100 GPUs supports the need for this kind of tool. Hardware failures hit under 1% of jobs but consumed 19% of GPU runtime. The study also projects a mean time to failure of 1.8 hours for 16,384-GPU jobs. [3]
Meta's cluster data show that large, long jobs bear a disproportionate share of failure costs. If the tool performs as described, Slurm operators could cut idle time between failure and restart.
Read the full assessment
Actual savings depend on each cluster's own failure rates.
Executive brief
NVIDIA has open-sourced NVIDIA Recovery under Apache 2.0. It is a cluster-level service that detects crashed, hung or slow training jobs, excludes suspect nodes and restarts the jobs from checkpoint. NVIDIA's blog presents it as production tooling, but the project's own GitHub README calls it experimental and not ready for production use. NVIDIA reports three months of data from one cluster: about 8,550 anomaly detections, a mean of 69 seconds to detect a problem and 94 seconds to act on it (NVIDIA blog). No independent evaluation was found. It currently supports Slurm only.
What changed and event timeline
(year unstated, presumably 2026): Production monitoring window
On one NVIDIA training cluster covering nearly 200 jobs a day, NVIDIA logged about 8,550 detections, with mean detection and action latencies of 69 and 94 seconds ().
Meta quantifies the cost of failures
Meta FAIR studied 11 months of data from clusters with 24,000 A100 GPUs. Hardware failures hit under 1% of jobs but 19% of GPU runtime, because large, long jobs are the most exposed ().
Aggregator pickup
Coverage so far, such as, restates NVIDIA's figures without adding new data.
Open-source release announced
The release is part of DSX OS. It ships as a Helm chart with a reference Slurm "Advisor Controller". IBM Research and Verda are named as early adopters (;).
Capabilities and access
- Repository and license: dsx-ai-factory/NVIDIA-Recovery, Apache 2.0. Some vendored pieces are MIT-licensed, and the MinIO dependency is AGPL v3.
- Hardware and schedulers: Ampere through Blackwell GPUs. Supports Slurm, including Slurm on Kubernetes via Slinky's slurm-operator.
- Dependencies: a Loki or VictoriaLogs log backend, Prometheus and MySQL.
Read the full section
- Repository and license: dsx-ai-factory/NVIDIA-Recovery, Apache 2.0. Some vendored pieces are MIT-licensed, and the MinIO dependency is AGPL v3. No version number is given; the project is labeled experimental.
- Hardware and schedulers: Ampere through Blackwell GPUs. Supports Slurm, including Slurm on Kubernetes via Slinky's slurm-operator. Native Kubernetes recovery (JobSet, Kubeflow PyTorchJob) is on the roadmap (FAQ).
- Dependencies: a Loki or VictoriaLogs log backend, Prometheus and MySQL.
- Opt-in: each job opts in with a single
#SBATCH --commentJSON directive. - NVIDIA Attribution (optional diagnosis add-on): its source code is not publicly fetchable (GitHub).
Technical analysis for researchers and developers
- Four-stage loop:
- Detect: log patterns for PyTorch/NCCL, CUDA and interconnect failures. Missing progress signals for hangs. Sustained throughput drop against a recent baseline for stragglers.
- Attribute: classify the failure.
Read the full section
- Four-stage loop:
- Detect: log patterns for PyTorch/NCCL, CUDA and interconnect failures; missing progress signals for hangs; sustained throughput drop against a recent baseline for stragglers.
- Attribute: classify the failure.
- Isolate: exclude implicated nodes and track them across restart attempts.
- Restart or stop: act through an HTTP/gRPC Advisor API.
- Scheduler-neutral design: a cluster-side controller translates recommendations into scheduler actions. Recommendations are recorded before execution, and the controller re-checks scheduler state after its own restart so it does not repeat actions.
- Watchdog: releases jobs to normal scheduling if a recovery decision stalls.
- What it does not do: create or restore checkpoints, or repair hardware (FAQ).
- What the 94-second figure covers: time until the automated action, not until training resumes.
Claims and evidence
- About 8,550 detections, 69 s / 94 s mean latencies: reported by NVIDIA from its own monitoring of a single cluster (blog).
- Use across "more than 100,000 GPUs": appears only in an author bio, describing the reference implementation inside Mission Control and DGX Cloud.
- IBM Research used monitor mode, then turned on autonomous action: NVIDIA's account; no IBM statement was found.
Read the full section
- About 8,550 detections, 69 s / 94 s mean latencies: reported by NVIDIA from its own monitoring of a single cluster (blog). There is no published false-positive rate, recovery success rate or goodput gain, and no independent corroboration.
- Use across "more than 100,000 GPUs": appears only in an author bio, describing the reference implementation inside Mission Control and DGX Cloud.
- IBM Research used monitor mode, then turned on autonomous action: NVIDIA's account; no IBM statement was found.
- Failure-exposure rationale: independently supported by Meta's study. It projects a mean time to failure of 1.8 hours for 16,384-GPU jobs.
Context and prior work
- Layered stack: NVSentinel handles node health on Kubernetes, Slurm prolog/epilog scripts handle node checks, and NVRx handles in-job restart, checkpointing and straggler detection.
- Lineage:
- Product lead Ian Gibbs previously helped build AWS SageMaker HyperPod, which covers similar resilient-training ground.
Read the full section
- Layered stack: NVSentinel handles node health on Kubernetes, Slurm prolog/epilog scripts handle node checks, and NVRx handles in-job restart, checkpointing and straggler detection. Recovery fills the gap across jobs.
- Lineage:
- Product lead Ian Gibbs previously helped build AWS SageMaker HyperPod, which covers similar resilient-training ground.
- Architect Dong Ahn led the Flux scheduler at Lawrence Livermore National Laboratory.
- Prior evidence: Meta's reliability taxonomy is the main independent baseline.
Limitations, safety and contested findings
- Maturity conflict: the blog's production framing and named adopters sit uneasily with the README and FAQ, which say interfaces may change without compatibility guarantees and a working local demo does not establish production readiness.
- No guarantees: the FAQ does not promise every failure is diagnosed or every retry succeeds. Hangs and slowdowns go undetected without the right telemetry.
- Detection numbers lack precision data: the 8,550 figure counts detections, not confirmed incidents, so its signal-to-noise ratio is unknown.
Read the full section
- Maturity conflict: the blog's production framing and named adopters sit uneasily with the README and FAQ, which say interfaces may change without compatibility guarantees and a working local demo does not establish production readiness.
- No guarantees: the FAQ does not promise every failure is diagnosed or every retry succeeds. Hangs and slowdowns go undetected without the right telemetry.
- Detection numbers lack precision data: the 8,550 figure counts detections, not confirmed incidents, so its signal-to-noise ratio is unknown.
- Safety default: autonomous action is off out of the box; deployments start in monitor mode.
Business and practitioner implications
- Who benefits now: Slurm shops running many multi-node jobs can trial it at low risk in monitor mode, comparing its advice with their engineers' manual decisions.
- Prerequisites: checkpoint-resumable workloads, centralized logs and a MySQL service.
- Who must wait: Kubernetes-native teams.
Read the full section
- Who benefits now: Slurm shops running many multi-node jobs can trial it at low risk in monitor mode, comparing its advice with their engineers' manual decisions.
- Prerequisites: checkpoint-resumable workloads, centralized logs and a MySQL service.
- Who must wait: Kubernetes-native teams.
- Licensing and dependence: review the AGPL dependency, and note that the deepest diagnosis relies on NVIDIA Attribution, whose source isn't public.
- Business case: sizing savings needs your own failure rates. NVIDIA has published no goodput or cost-savings figure.