InfrastructureCompute, chips & cloud
NVIDIA and Princeton's ReplaySSM stores a checkpoint and recent inputs instead of full SSM state to speed up hybrid-model decoding
ReplaySSM, from NVIDIA and Tri Dao's Princeton lab, stops rewriting the full state-space-model state on every decoding step. It keeps a checkpoint plus a short buffer of recent inputs, which speeds up large-batch decoding of hybrid models. Parts are merged into vLLM and TensorRT-LLM.
From the NVIDIA Generative AI article of October 12, 2026 and cited sources: In normal steps, the method computes each output from a stored checkpoint plus a buffer of up to 16 small per-token records, and writes only the new record. [1] [2]
SOURCES[1] NVIDIA blog: ReplaySSM developer.nvidia.com[2] arXiv 2610.13778 arxiv.org
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- In normal steps, the method computes each output from a stored checkpoint plus a buffer of up to 16 small per-token records, and writes only the new record. [1] [2]
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 12, 2026 article published
- Covered
- Oct 13 edition
- Evidence
- 10 sources across 4 sites
Reach: not measured yet
In normal steps, the method computes each output from a stored checkpoint plus a buffer of up to 16 small per-token records, and writes only the new record. Every 16 steps it folds the buffer into the checkpoint. This roughly halves state memory traffic, because checkpoint reads stay the same while full-state writes mostly disappear. [1] [2]
According to the authors' own benchmarks, at batch size 256 a full decode step runs 1.20–1.47× faster than a tuned vLLM baseline. There is no gain at small batches: vLLM's tests found about 0.96–1.05× at batch sizes 1–8. Speculative decoding also benefits, because discarding rejected draft tokens becomes a simple buffer-pointer move. The paper reports about 1.8× higher throughput at batch 256, and other setups report different figures. [1] [2] [5]
A high-precision checkpoint drives the recurrence while a low-precision copy serves reads, so rounding errors from reads don't build up. With 4-bit NVFP4 reads, three of four models stay within about 2 points of FP32 on GPQA-Diamond. Nemotron-3-Nano-4B loses 5.6–6.7 points. [1]
The code is open source. vLLM merged opt-in Mamba-2 support for standard decoding only, and TensorRT-LLM merged a replay variant for Mamba multi-token prediction. A competing method, SketchSSM, notes that ReplaySSM still reads the full state on every step and reports kernels 3.24–3.69× faster than ReplaySSM. [4] [5] [7] [8]
at large serving batches, SSM layers can take most of a hybrid model's decode time, and cutting state writes reportedly speeds decoding.
Read the full assessment
Implication: teams serving Nemotron- or Qwen3.5-style hybrids at high concurrency may lower costs, but small-batch workloads see no gain.
Executive brief
In hybrid language models (Nemotron 3, Qwen3.5), the state-space model (SSM) layers, not attention, can take the largest share of each decoding step at serving batch sizes, up to 100K tokens of context. That is NVIDIA's own measurement (NVIDIA blog). ReplaySSM, from NVIDIA and Tri Dao's lab at Princeton, stops writing the full SSM state on every token. It keeps a checkpoint plus a short buffer of recent inputs. At batch size 256 this makes decoding 1.20–1.47× faster than a tuned vLLM baseline (arXiv). The code is open source and parts of it are already merged into vLLM and TensorRT-LLM. No independent replication has been found.
What changed and event timeline
Dao AI Lab preview
An earlier version of the method was described on Tri Dao's blog. It reported up to 1.43× faster standard decoding and 3.0–3.3× more concurrent requests, on models up to 550B parameters ().
TensorRT-LLM merge
A checkpoint-based "replay" variant for Mamba multi-token prediction (MTP) was merged. The PR says it was faster in 39 of 44 batch-size/data-type configurations ().
vLLM Mamba-2 support merged
This followed a July 20 RFC. The feature is opt-in and covers only standard decoding, not speculative decoding (;).
Competing method claims to beat it
SketchSSM (v2) points out that ReplaySSM still reads the full state on every step. It reports its own linear-attention kernels running 3.24–3.69× faster than ReplaySSM ().
Paper posted
"ReplaySSM: Cache SSM Inputs, Not State" (v1) appeared on arXiv, with seven authors from NVIDIA and Princeton ().
NVIDIA explainer
The blog post sums up the speed, speculative decoding and quantization results. It also says the method runs in production in GLM-5.3-Flash and Kimi-K3 ().
Capabilities and access
- Layer types: Mamba-2 and Gated DeltaNet (GDN) Triton kernels for standard and speculative decoding.
- Code: a vLLM fork (
0.23.1+replayssm) enabled with flags--use-replayssm,--use-replayssm-specand--replayssm-buffer-len 16(GitHub). - Upstream: the vLLM merge requires
--mamba-cache-mode none(PR 48018). NVIDIA also lists PRs for FlashInfer and SGLang (NVIDIA blog).
Read the full section
- Layer types: Mamba-2 and Gated DeltaNet (GDN) Triton kernels for standard and speculative decoding. Low-precision CuTe kernels (FP16 with stochastic rounding, FP8, NVFP4) target the B300 GPU (GitHub).
- Code: a vLLM fork (
0.23.1+replayssm) enabled with flags--use-replayssm,--use-replayssm-specand--replayssm-buffer-len 16(GitHub). - Upstream: the vLLM merge requires
--mamba-cache-mode none(PR 48018). NVIDIA also lists PRs for FlashInfer and SGLang (NVIDIA blog).
Technical analysis for researchers and developers
- Core idea: the recurrence is unrolled over a window of L=16 tokens. A "flush" every L steps folds the buffer into the checkpoint.
- Speculative decoding: rolling back rejected drafts becomes a buffer-pointer move.
- Quantization: a high-precision checkpoint drives the recurrence while a low-precision copy serves reads, so read rounding error does not compound.
Read the full section
- Core idea: the recurrence is unrolled over a window of L=16 tokens. Most steps compute the output from the checkpoint plus buffered (decay, value, key) records and write only that small record. A "flush" every L steps folds the buffer into the checkpoint. Checkpoint reads stay the same, so total state traffic is roughly halved (NVIDIA blog).
- Speculative decoding: rolling back rejected drafts becomes a buffer-pointer move. Verification uses "output-only evaluation", the same chunkwise matrix products used in prefill, so no intermediate state is built.
- Quantization: a high-precision checkpoint drives the recurrence while a low-precision copy serves reads, so read rounding error does not compound.
- Correctness checks: tests compare against an FP32 reference (rtol 1e-4) and check engine-level parity with greedy decoding at tensor-parallel sizes 1 and 2 (PR 48018).
Claims and evidence
All figures below were reported by the authors or in their own PRs. No third-party benchmark has been found.
Read the full section
All figures below were reported by the authors or in their own PRs. No third-party benchmark has been found.
- Standard decoding at batch 256: kernel 1.46–1.80× faster, full decode step 1.20–1.47× (NVIDIA blog). At batch 1 there was no speedup.
- Speculative decoding: the blog reports 1.85× and 2.10× vLLM's throughput at batch 512 on the two 120B-class models. The paper abstract says "about 1.8×" at batch 256 (arXiv), and the RFC reports up to 2.21× (RFC). These numbers come from different setups and should not be pooled.
- Accuracy: with NVFP4 state reads and an FP16 or FP8 recurrence, three of four models stay within about 2 points of FP32 on GPQA-Diamond. Nemotron-3-Nano-4B still loses 5.6–6.7 points (NVIDIA blog).
Context and prior work
- The method builds on Mamba-2, GDN and Kimi Delta Attention (KDA), and on the periodic state checkpointing explored in the Nemotron 3 Ultra report.
- Two parallel lines of work address SSM-state costs differently. DASC compresses stored state checkpoints and cites ReplaySSM as aimed at decode-time state traffic. SketchSSM cuts state reads, which ReplaySSM leaves in place.
Limitations, safety and contested findings
- Small batches: speedups appear only at large batch sizes. vLLM measured about 0.96–1.05× at batch 1–8 (PR 48018).
- Narrow evaluation: four models from two families, each on a single GPU without tensor parallelism. KDA results are not in the paper's evaluation.
- Contested speed: SketchSSM claims much larger speedups than ReplaySSM (SketchSSM), but that is a competitor's measurement.
Read the full section
- Small batches: speedups appear only at large batch sizes. vLLM measured about 0.96–1.05× at batch 1–8 (PR 48018).
- Narrow evaluation: four models from two families, each on a single GPU without tensor parallelism. KDA results are not in the paper's evaluation.
- Contested speed: SketchSSM claims much larger speedups than ReplaySSM (SketchSSM), but that is a competitor's measurement.
- Production claims: the GLM-5.3-Flash and Kimi-K3 deployments are stated only by NVIDIA. The Kimi-K3 arXiv abstract does not mention ReplaySSM (Kimi K3).
Business and practitioner implications
- Serving cost: for high-concurrency serving of hybrid models, the SSM layers are now the place to look for gains.
- Low-precision state: keeping a separate high-precision checkpoint makes 4-bit (NVFP4) state reads practical, which matters on Blackwell GPUs.
- Before adopting: run your own benchmarks at your actual batch sizes.
Read the full section
- Serving cost: for high-concurrency serving of hybrid models, the SSM layers are now the place to look for gains. Speculative decoding becomes worthwhile for these models where it previously barely helped.
- Low-precision state: keeping a separate high-precision checkpoint makes 4-bit (NVFP4) state reads practical, which matters on Blackwell GPUs.
- Before adopting: run your own benchmarks at your actual batch sizes. Also note that speculative decoding and GDN support were outside the first vLLM merge.