InfrastructureCompute, chips & cloud
NVIDIA combines Humming kernels, an FP8 LM head and MTP speculative decoding to speed Ling-3.0-flash on DGX Spark
An NVIDIA guide combines Humming MoE kernels, an FP8 LM head and multi-token-prediction speculative decoding to run Ant Group's 124B Ling-3.0-flash faster on DGX Spark: 185% on a synthetic test, 42.5% on GSM8K, with accuracy essentially unchanged.
From the NVIDIA Generative AI article of October 27, 2026 and cited sources: At batch size 1, decoding on DGX Spark is limited by memory bandwidth, so the gains add up when you cut both the bytes read per token and the number of decode steps. [1] [3]
SOURCES[1] 3 Ways to Boost MoE Inference on NVIDIA DGX Spark… developer.nvidia.com[3] Announcing Humming (vLLM Blog) vllm.ai
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- At batch size 1, decoding on DGX Spark is limited by memory bandwidth, so the gains add up when you cut both the bytes read per token and the number of decode steps. [1] [3]
- Original source
- NVIDIA Generative AI article, company source
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 27, 2026 article published
- Covered
- Oct 28 edition
- Evidence
- 7 sources across 7 sites
- 13 Ways to Boost MoE Inference on NVIDIA DGX Spark…developer.nvidia.com
- 2Ling-3.0-flash: Open-Weight 124B MoE Scores 38 on AA Index…orcarouter.ai
- 3Announcing Humming (vLLM Blog)vllm.ai
- 4Ling cookbook DGX Spark Humming notebookgithub.com
- 5inclusionAI/Ling-3.0-flash (Hugging Face)huggingface.co
- 6Which Inference Engine Wins on the DGX Spark (Medium)medium.com
Reach: not measured yet
At batch size 1, decoding on DGX Spark is limited by memory bandwidth, so the gains add up when you cut both the bytes read per token and the number of decode steps. Humming compiles kernels at runtime for the model's exact matrix shapes and added 9.3%. Moving the large LM head from BF16 to FP8 halves its weight reads, bringing the total to 21.7%. Adding the model's built-in NEXTN multi-token-prediction draft layer brought the synthetic gain to 185%, or about 103 tok/s. [1] [3]
The gain was much smaller on a real task. Across all 1,319 GSM8K questions, decode throughput rose from about 38 to 54 tok/s and mean latency fell from 15.2 to 11.5 seconds. Accuracy held at roughly 96.9%. The speculative-decoding speedup shrank because fewer draft tokens were accepted per round: 2.68 versus 3.90. The synthetic test inflated that number with repetitive, greedy, fixed-length outputs. [1]
Turning on the FP8 LM head and MTP together without a code fix fails silently. Unless the draft model shares the whole LM-head module, it reads the FP8 weights as BF16, accepts only one token per round, and ends up slower than the unoptimized baseline. No error is raised. That makes a startup check that each optimization is actually active worth adding. [1]
a 124B open-weight MoE reached about 54 tok/s on real math prompts on a desktop system.
Read the full assessment
Implication: local deployment for data-residency-bound teams looks more practical, but speedups should be measured on actual workloads and each optimization verified at startup.
Executive brief
The biggest single gain in the post, 185% faster decoding, does not hold up on a real task. On NVIDIA DGX Spark, three stacked optimizations sped up decoding of Ant Group/inclusionAI's Ling-3.0-flash by 185% on a synthetic benchmark. On the full GSM8K math test set the gain was 42.5%.
Read the full section
The biggest single gain in the post, 185% faster decoding, does not hold up on a real task. On NVIDIA DGX Spark, three stacked optimizations sped up decoding of Ant Group/inclusionAI's Ling-3.0-flash by 185% on a synthetic benchmark. On the full GSM8K math test set the gain was 42.5%. The post also found a hidden failure: when two of the optimizations are combined without a specific code fix, the system silently runs slower than the unoptimized baseline (NVIDIA Technical Blog). The results come from a single NVIDIA solutions architect's post. No independent replication has been found yet. The practical lessons are to benchmark on your real workload and to check at startup that each optimization is actually on.
What changed and event timeline
Ling-3.0-flash announced
Ant Group's inclusionAI released a 124B-total, 5.1B-active hybrid MoE. Weights followed on Aug 7 under the MIT license, according to a third-party summary ().
Humming kernels open-sourced
Ant Group and the vLLM community announced Humming, a JIT kernel library for quantized inference that ships as a dependency in vLLM 0.29.0 ().
NVIDIA publishes the DGX Spark tuning guide
The post covers swapping in Humming as the MoE backend, FP8 quantization of the LM head, and NEXTN multi-token-prediction (MTP) speculative decoding for Ling-3.0-flash in SGLang ().
Cookbook notebook linked
The deployment recipe for the combined setup lives in inclusionAI's Ling cookbook ().
Capabilities and access
- Model tested: Ling-3.0-flash-fp4, 124B total and 5.1B active parameters, run at batch size 1 on DGX Spark (NVIDIA).
- Architecture: 35 Kimi Delta Attention (KDA) linear-attention layers plus 7 MLA layers.
- Context and license: 256K native context, MIT license (Hugging Face card).
Read the full section
- Model tested: Ling-3.0-flash-fp4, 124B total and 5.1B active parameters, run at batch size 1 on DGX Spark (NVIDIA).
- Architecture: 35 Kimi Delta Attention (KDA) linear-attention layers plus 7 MLA layers. Each token is routed to 8 of 512 experts plus one shared expert (Hugging Face card).
- Context and license: 256K native context, MIT license (Hugging Face card).
- Hardware: DGX Spark uses the GB10 chip with 128 GB of unified LPDDR5x memory at 273 GB/s (Medium/Hannecke).
- Software:
pip install humming-kernels(PyPI). FP8 LM head is turned on withSGLANG_ENABLE_FP8_LM_HEAD=1. The post does not list SGLang or CUDA versions.
Technical analysis for researchers and developers
- Bottleneck: at batch size 1, decoding was limited by memory bandwidth.
- Humming: compiles kernels at runtime (via NVRTC) for the model's exact matrix shapes, including its narrow 768 MoE intermediate size.
- FP8 LM head: the checkpoint excludes the LM head from quantization, so it is quantized on the fly at load time.
Read the full section
- Bottleneck: at batch size 1, decoding was limited by memory bandwidth. Reading the MoE weights and the LM head (2,560 × 157,184) dominated each step.
- Humming: compiles kernels at runtime (via NVRTC) for the model's exact matrix shapes, including its narrow 768 MoE intermediate size. Each kernel compiles in seconds; FlashInfer's CUTLASS path builds 97 targets and takes tens of minutes.
- FP8 LM head: the checkpoint excludes the LM head from quantization, so it is quantized on the fly at load time. Moving its weights from BF16 to FP8 halves the bytes read per token. This option cannot be combined with
--enable-fp32-lm-head. - MTP: the model's built-in NEXTN layer acts as the draft model, with 3 draft steps and a 4-token verification window. One verification step costs about 2.26× a normal decode step, mainly because checking 4 tokens can activate up to 32 experts.
- Combination bug: with FP8 and MTP both on, the draft model must share the whole LM-head module, not just its weight tensor. Otherwise it reads FP8 weights as BF16 and accepts only 1.00 token per round.
Source for this section: NVIDIA.
Claims and evidence
All of these numbers come from NVIDIA alone and have not been independently replicated.
Read the full section
All of these numbers come from NVIDIA alone and have not been independently replicated.
- Synthetic test (32K input, 2K output, greedy decoding):
- Baseline: 27.71 ms per output token (36.09 tok/s).
- Humming: +9.3%.
- Humming + FP8 LM head: +21.7%.
- All three: 102.88 tok/s (+185.1%), with 3.90 tokens accepted per round (NVIDIA).
- GSM8K, all 1,319 questions, 5-shot, temperature 0.6:
- Decode throughput: 37.81 → 53.89 tok/s.
- Mean latency: 15.21 → 11.49 s.
- Accuracy: 96.89% → 96.82%.
- Accept length: 2.68 tokens per round (NVIDIA).
- The post says the accuracy change is "not statistically detectable" but does not name the test it used.
- Humming's own claims (up to 5× kernel speedups; 14–43% end-to-end gains on 8× H20) come from its developers (vLLM Blog).
Context and prior work
- Humming targets GPUs from Turing (SM75) onward.
- Artificial Analysis gives Ling-3.0-flash an Intelligence Index score of 38 and measured about 368 tok/s on hosted APIs, as reported by OrcaRouter.
- A community recipe for the Ling-3.0-flash-VL FP8 variant on two DGX Sparks reports about 26 tok/s for a single request without speculative decoding (Context Studios).
Read the full section
- Humming targets GPUs from Turing (SM75) onward. It includes a fast path that converts MXFP4 weights to INT8/FP8 for GPUs without native MXFP4 support (vLLM Blog).
- Artificial Analysis gives Ling-3.0-flash an Intelligence Index score of 38 and measured about 368 tok/s on hosted APIs, as reported by OrcaRouter.
- A community recipe for the Ling-3.0-flash-VL FP8 variant on two DGX Sparks reports about 26 tok/s for a single request without speculative decoding (Context Studios). It uses a different model variant and setup, so it is not directly comparable.
Limitations, safety and contested findings
- The author notes that repetitive synthetic outputs, greedy sampling and a fixed 2K-token output length inflate the speculative-decoding gain.
- The combination bug degrades performance below the baseline without any error message.
- All results are for one model at batch size 1 and one real benchmark (GSM8K). Software versions are not listed.
Read the full section
- Synthetic gains overstate real ones. The author notes that repetitive synthetic outputs, greedy sampling and a fixed 2K-token output length inflate the speculative-decoding gain. On GSM8K the throughput gain fell from 185% to 42.5%.
- Silent failure. The combination bug degrades performance below the baseline without any error message.
- Narrow scope. All results are for one model at batch size 1 and one real benchmark (GSM8K). Software versions are not listed.
- Small extra startup delay. Mean time to first token rose slightly, from 0.207 s to 0.214 s.
Source for this section: NVIDIA.
Business and practitioner implications
- A 124B open-weight MoE running at about 54 tok/s on a desktop box makes local agent and coding deployments more realistic for teams with data-residency requirements.
- On bandwidth-limited hardware, cutting bytes read per token (custom kernels, a quantized LM head) and cutting the number of decode steps (MTP) compound.
- Speedup figures from vendor synthetic tests should not drive purchasing decisions.
Read the full section
- A 124B open-weight MoE running at about 54 tok/s on a desktop box makes local agent and coding deployments more realistic for teams with data-residency requirements.
- On bandwidth-limited hardware, cutting bytes read per token (custom kernels, a quantized LM head) and cutting the number of decode steps (MTP) compound. NVIDIA's numbers show this.
- Budget for real-workload benchmarks. Speedup figures from vendor synthetic tests should not drive purchasing decisions.
- Add startup checks that confirm each optimization is active, rather than inferring it from throughput numbers.
- Humming's support in both vLLM and the Ling SGLang recipe lowers the effort of trying it.