DeepSeek Open-Sources DeepInfer, the Engine Behind DeepSeek-V4.1-Flash

DeepSeek has open-sourced DeepInfer, the inference framework that runs its production services, and it says a single cluster running the stack serves 155 trillion tokens per day. The engine powers DeepSeek-V4.1-Flash and ships under the Apache 2.0 license on GitHub. Alongside it, DeepSeek released deepseek-recipe, its MIT-licensed token encoding and decoding layer.

For developers, this is a rare look inside a frontier lab's real serving stack. DeepSeek built DeepInfer primarily on vLLM, with ideas borrowed from SGLang and TensorRT-LLM. The release covers offline inference with the core features DeepSeek uses at scale.

The basics from the model card and README:

• Model support: one model, DeepSeek-V4.1-Flash, with a 1M-token context window
• Hardware: NVIDIA SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs
• Benchmarks: 4 GPUs for prefill, 64 for decode
• Economics: DeepSeek reports revenue of US$38 per GPU-hour at valley pricing

That last number matters. Serving economics decide whether open-weight models can compete with closed APIs, and DeepSeek just published its unit economics alongside the code.

How does DeepInfer actually work?

The engine stacks DeepSeek's own kernels — FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels — and puts five system-level ideas on top.

Attention DP plus Wide-EP. Attention runs under data parallelism; the MoE layers use expert parallelism with an expert-parallel load balancer (EPLB). Decode runs EP64 to EP128 inside a supernode. Prefill uses the smallest expert-parallel degree that memory allows, typically EP2 to EP8.

DSpark, confidence-scheduled speculative decoding. A draft model proposes 5 tokens per step. A Confidence Head predicts each draft token's acceptance rate, and the scheduler verifies only the most valuable prefix. Per the deployment Q&A, this cuts verified draft tokens by about 30% (from 5.00 down to between 3.2 and 4.4) while accepted length barely moves — 3.29 versus 3.20. In short: less verification work for the same output.

SWA bounded replay. On a cache hit, DeepInfer recomputes only the last 128 tokens of sliding-window attention state. That makes recomputation cost constant regardless of context length. In the decoder layers, DeepSeek says this nearly halves total prefill compute.

Asynchronous scheduling, determinism and structured output. CPU batch scheduling overlaps with GPU execution so the GPU never idles between steps. Batch-invariant inference and deterministic sampling are available on NVIDIA only. Structured outputs support regex, JSON schema and EBNF grammars.

The takeaway: DeepInfer's speed comes from stacking several small efficiency tricks, not one big breakthrough.

How fast is it?

DeepSeek published Pareto curves for DeepSeek-V4.1-Flash at a 131,072-token context. All figures are self-reported and read from those charts.

• Decode (EP64DP64, nextn=3, 75% acceptance): above 25,000 tokens/s/GPU at 256 requests per rank, near 100 tokens/s per user. At 8 requests per rank, per-user speed passes 300 tokens/s.
• Prefill (EP4DP4, 140-token bounded replay): about 60,000 non-cached tokens/s/GPU at 4,096 tokens per rank. Throughput nears 90,000 at 65,536 tokens, at roughly 730 ms per step.
• Wide-EP payoff: the EP8 versus EP64 chart shows EP64 ahead at every batch size.

DSpark's dynamic verification adds 4% to 7% throughput at about 60 sequences per rank. At 16 sequences per rank it's 1.5% slower than fixed 5-token verification — so the win depends on load.

The clear pattern: throughput scales with batch size and expert-parallel width, while per-user latency falls as concurrency rises.

How does it compare with vLLM, SGLang and TensorRT-LLM?

Feature	DeepInfer	vLLM	SGLang	TensorRT-LLM
Maintainer	DeepSeek	vLLM project	SGLang project	NVIDIA
License	Apache 2.0	Apache 2.0	Apache 2.0	Apache 2.0
Model coverage	1 (V4.1-Flash)	200+ architectures	LLMs, VLMs, diffusion	Support matrix
Hardware	SM100, Ascend 950	NVIDIA, AMD, Intel, CPUs, plugins	NVIDIA, AMD, TPU, Intel, Apple, Ascend	NVIDIA GPUs
Expert parallelism	Yes, with EPLB	Yes	Yes	Wide-EP
Speculative decoding	DSpark (confidence-scheduled)	n-gram, suffix, EAGLE, DFlash	Yes, via SpecForge	Yes
P/D disaggregation	Not in this release	Yes	Via llm-d, Dynamo	Yes
Online serving	Offline only	OpenAI-compatible server	Yes	Yes

The comparison is blunt: DeepInfer trades breadth for depth. One model, two hardware targets, but a serving stack tuned to that exact combination.

What's missing?

A full service still needs a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. The DeepSeek team states those components are tightly coupled to internal infrastructure, so they stayed home.

The hardware requirements are narrow too. Builds need Linux, GCC 13 or later, Python 3.12, and CUDA 13 or CANN 9.2. The smallest efficient production unit is 64 prefill GPUs plus 64 decode GPUs — 128 accelerators before the stack hits its stride.

One more wrinkle: the internal FlashMLA decoding kernel runs 5% to 10% faster than the public one. DeepSeek puts the end-to-end gap at 5% or less.

Why does this release matter?

Inference cost, not training cost, now shapes who can compete in AI. By open-sourcing the engine that serves 155 trillion tokens a day at US$38 per GPU-hour, DeepSeek has shown its full production recipe — or most of it. The missing KV cache, disaggregation and load-balancing layers, plus the 128-GPU minimum, mean most teams will read this code rather than run it. Either way, the techniques — DSpark's confidence scheduling, SWA bounded replay, wide expert parallelism — will likely show up in vLLM and SGLang forks within months.

By Sophie Lindqvist