InfrastructureCompute, chips & cloud
DeepSeek open-sources DeepInfer, the inference engine behind its API
DeepSeek released DeepInfer, the engine serving its API, under Apache 2.0. It runs only V4.1-Flash on Blackwell or Ascend 950 hardware, withholds key cluster components, and early third-party tests disagree on whether it outperforms vLLM.
From the r/AISEOInsider (via last30days) article of October 22, 2026 and cited sources: DeepInfer supports a single model, DeepSeek-V4.1-Flash, and only Nvidia Blackwell (SM100) or Huawei Ascend 950 chips. [5]
SOURCES[5] SiliconANGLE: DeepSeek open-sources DeepInfer siliconangle.com
AT Pulse research deskPublished · Updated · 5 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- DeepInfer supports a single model, DeepSeek-V4.1-Flash, and only Nvidia Blackwell (SM100) or Huawei Ascend 950 chips. [5]
- Original source
- r/AISEOInsider (via last30days) article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- When
- October 22, 2026 article published
- Covered
- Oct 22 edition
- Evidence
- 13 sources across 11 sites
- 1Free DeepSeek Inference Engine Runs 155 Trillion Tokens a…reddit.com
- 2DeepSeek DeepInfer Just Changed AIreddit.com
- 3Linked video: Free DeepSeek Inference Engine Runs 155…youtube.com
- 4BigGo Financefinance.biggo.com
- 5SiliconANGLE: DeepSeek open-sources DeepInfersiliconangle.com
- 6CO/AI: DeepSeek pivots to sharing componentsgetcoai.com
RisingHow it’s measured
YouTube video · 16.5K views · 177 likes
Reach blends public engagement (views, likes, comments, shares, stars) from the platforms where the story or its original source lives: YouTube, Hacker News, GitHub, Substack and similar. A video and the article written from its transcript share one score, taken from the video. Platforms with no data are left out rather than counted as zero, and the age of a story never lowers its score.
DeepInfer supports a single model, DeepSeek-V4.1-Flash, and only Nvidia Blackwell (SM100) or Huawei Ascend 950 chips. It omits the KV-cache system, prefill/decode disaggregation and load-balancing code, and its smallest efficient setup is 64 prefill plus 64 decode GPUs. [5]
Two techniques target cost. DSpark speculative decoding adds a confidence head that predicts which drafted tokens will be accepted, cutting verified draft tokens by about 30% for a 4–7% throughput gain. Bounded replay reprocesses only the last 128 prompt tokens through sliding-window layers, nearly halving prefill compute. vLLM measured quality gaps within about 1.5 standard errors on GSM8K and GPQA. [5] [7]
The headline 155 trillion tokens a day from one cluster is a DeepSeek-reported figure that counts cache-hit input tokens. It rests on a 96% cache-hit rate, so it does not mean 155 trillion newly generated tokens. [4]
The engine packages production techniques for prefill-heavy agent workloads, and deepseek-recipe aims to make third-party hosts behave like the official API. In practice, the full benefits favor operators running 128 or more GPUs.
Read the full assessment
Single-node teams may still do better on vLLM.
Research is done; writing the dossier now.
Executive brief
DeepSeek has open-sourced DeepInfer, the inference engine behind its API, under Apache 2.0. In the same release, the company said a single serving cluster handles 155 trillion tokens a day. That figure counts cache-hit input tokens, so it does not mean 155 trillion newly generated tokens (BigGo Finance). DeepInfer runs only one model, DeepSeek-V4.1-Flash, and only on Nvidia Blackwell or Huawei Ascend 950 hardware. It leaves out the KV-cache, disaggregation and load-balancing code (SiliconANGLE). Early third-party benchmarks disagree on whether it beats vLLM. The answer depends on deployment scale.
What changed and event timeline
DeepSeek declines a full release
DeepSeek said its engine had drifted far from vLLM and was tied to internal infrastructure. It planned to contribute modular components to existing projects instead ().
vLLM tunes V4.1-Flash
vLLM reported a 5.3× throughput gain under a 150 tokens/s per-user constraint. It credited bounded replay and DeepSeek kernels such as MegaAttention ().
V4.1-Flash-1017 ships
DeepSeek says the update finishes tasks faster. In its own table, the model scores 77% on DeepSWE v1.1 ().
DeepInfer open-sourced
The release landed one day after the US Department of Homeland Security (DHS) reportedly ordered federal agencies to remove DeepSeek products ().
First third-party benchmarks
On a single 8×B300 node, vLLM beat DeepInfer (). On 8×B200, DeepInfer led on throughput ().
Capabilities and access
- Model: only DeepSeek-V4.1-Flash. The default build is 1017 (SiliconANGLE).
- Hardware: Nvidia SM100 (Blackwell) and Huawei Ascend 950. Hopper and older Nvidia chips are not supported (same source).
- Features: structured outputs, batch-invariant inference and deterministic sampling. The linked post and the video say the last two work only on Nvidia (08:44).
Read the full section
- Model: only DeepSeek-V4.1-Flash. The default build is 1017 (SiliconANGLE).
- Hardware: Nvidia SM100 (Blackwell) and Huawei Ascend 950. Hopper and older Nvidia chips are not supported (same source). The video transcript says "Ascend 910" (09:15), which conflicts with written coverage.
- Features: structured outputs, batch-invariant inference and deterministic sampling. The linked post and the video say the last two work only on Nvidia (08:44).
- Minimum efficient setup: 64 GPUs for prefill plus 64 for decode (SiliconANGLE).
- Related tool: deepseek-recipe reproduces the official API's token processing so self-hosted output matches DeepSeek's (The Decoder).
Technical analysis for researchers and developers
- A draft model proposes 5 tokens. The video says "4% to 10%" (07:16).
- Prefill reruns only the last 128 tokens through sliding-window layers, which nearly halves prefill compute. vLLM describes this as trading exactness for efficiency.
- Vision encoding, prefill and decode are separate stages. Shared KV storage sits on 3FS (Latent Space).
Read the full section
- DSpark speculative decoding. A draft model proposes 5 tokens. A confidence head predicts which ones will be accepted, so only likely tokens get verified. This cuts verified draft tokens by about 30% and adds 4–7% throughput at production load (SiliconANGLE). The video says "4% to 10%" (07:16).
- SWA bounded replay. Prefill reruns only the last 128 tokens through sliding-window layers, which nearly halves prefill compute. vLLM describes this as trading exactness for efficiency. It measured quality gaps within about 1.5 standard errors on GSM8K and GPQA (vLLM).
- Serving design. Vision encoding, prefill and decode are separate stages. Shared KV storage sits on 3FS (Latent Space).
Claims and evidence
- Cluster numbers: 155T tokens/day, 96% cache-hit rate, about 400 ms time to first token, 230 tokens/s output, and $38 revenue per GPU-hour.
- Margins: the estimate of over 80% margin and an 81-day payback is commentary from Latent Space, not a DeepSeek figure.
- Reproduction: vLLM reportedly matched DeepInfer's decode throughput within 6% on GB200. That covers the decode stage only, not the full service (same source).
Read the full section
- Cluster numbers: 155T tokens/day, 96% cache-hit rate, about 400 ms time to first token, 230 tokens/s output, and $38 revenue per GPU-hour. All of these come from DeepSeek's report (BigGo).
- Margins: the estimate of over 80% margin and an 81-day payback is commentary from Latent Space, not a DeepSeek figure. It leaves out costs for the SSD cache fleet, networking and idle capacity (Latent Space).
- Reproduction: vLLM reportedly matched DeepInfer's decode throughput within 6% on GB200. That covers the decode stage only, not the full service (same source).
- Benchmarks: the DeepSWE scores are 77% for the new Flash, 74.2% for the September version and 74% for Claude Opus 5. All come from DeepSeek's own testing. Opus 5 still wins 7 of the 11 published benchmarks, including Humanity's Last Exam, 56.3 to 42.6 (SiliconANGLE).
- GitHub traction: the claim of 12,800 stars in a day appears only in the video (03:11) and the linked post. No other coverage confirms it.
Context and prior work
DeepInfer is built mostly on vLLM and borrows ideas from SGLang and TensorRT-LLM (video 11:36; The Decoder). It follows the 2025 decision to release components rather than a full engine (CO/AI). Observers read it mostly as production lessons feeding back into the vLLM ecosystem, not as a replacement for vLLM (Latent Space).
Limitations, safety and contested findings
- Incomplete release: the KV-cache system, the prefill/decode split and load balancing are withheld. A Zhihu post by DeepSeek explains how they work (SiliconANGLE).
- Contradictory benchmarks: on one 8×B300 node, vLLM was 2.0× faster on decode and 9× faster on prefill. Neither engine was fully tuned (GPUStack).
- Unverified hardware claims: DeepSeek's speed charts ran on Nvidia. Performance on Ascend has not been checked independently (video 11:27).
Read the full section
- Incomplete release: the KV-cache system, the prefill/decode split and load balancing are withheld. A Zhihu post by DeepSeek explains how they work (SiliconANGLE).
- Contradictory benchmarks: on one 8×B300 node, vLLM was 2.0× faster on decode and 9× faster on prefill. The author says DeepInfer is built for rack-scale, disaggregated deployment, so a single node is not its intended setup (DEV). On 8×B200, DeepInfer had about 25% higher output throughput, but vLLM's first token came about 2× faster. Neither engine was fully tuned (GPUStack).
- Unverified hardware claims: DeepSeek's speed charts ran on Nvidia. Performance on Ascend has not been checked independently (video 11:27).
Business and practitioner implications
- Who can use it: a 128-GPU minimum puts the full benefit within reach of cloud providers and large operators, not small teams.
- Consistency across hosts: deepseek-recipe should make Flash behave the same across third-party hosts, which matters for multi-vendor sourcing.
- Agent workloads: they are prefill-heavy, so bounded replay and high cache-hit rates are the levers to watch.
Read the full section
- Who can use it: a 128-GPU minimum puts the full benefit within reach of cloud providers and large operators, not small teams. Single-node users may do better on vLLM today.
- Consistency across hosts: deepseek-recipe should make Flash behave the same across third-party hosts, which matters for multi-vendor sourcing.
- Agent workloads: they are prefill-heavy, so bounded replay and high cache-hit rates are the levers to watch. The economics depend on a 96% cache-hit rate.
- Video framing: its suggestion to use Flash for bulk work and premium models for hard tasks (12:35) sits inside a course pitch.
- Compliance: US public-sector buyers face the reported DHS directive.
The source trail.
Sources (13)
Free DeepSeek Inference Engine Runs 155 Trillion Tokens a Day
Feed excerpt only. A transcript of the video linked from this page is supplied (linked video auto captions; https://www.youtube.com/watch?v=GL-nOc43omg); automatic text may contain errors.
reddit.comDeepSeek DeepInfer Just Changed AI
Related coverage; assess separately
reddit.comLinked video: Free DeepSeek Inference Engine Runs 155 Trillion Tokens a Day
linked video auto captions
www.youtube.com