DeepSeek Open-Sources DeepInfer, the Engine Behind 155T Tokens a Day

DeepSeek has open-sourced DeepInfer, the in-house inference framework it uses to serve DeepSeek V4.1 Flash in production. The code is on GitHub under the Apache 2.0 license, and DeepSeek released a companion library, deepseek-recipe, that handles the API-to-token plumbing behind its official API. DeepSeek says a single cluster running this stack serves 155 trillion tokens a day and brings in about $38 per GPU-hour at off-peak prices.

It was the biggest AI story on Hacker News in the last day, past 2,100 points and 800 comments, and DeepSeek’s announcement post on X passed 4 million views. The interest is easy to explain. Labs rarely publish the serving code that sets their real cost per token, and this release comes with a detailed deployment Q&A explaining how DeepSeek runs it.

Deployability and license: Apache 2.0 for DeepInfer, MIT for deepseek-recipe. But this is an offline inference engine, not a full serving stack. It supports one model (DeepSeek V4.1 Flash) on two hardware families (NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs). The KV cache system, prefill/decode disaggregation, and load balancing that make it a production service are not included. All performance and business figures below come from DeepSeek. 9to6AI has not built or benchmarked DeepInfer.

What shipped

• The engine: optimized kernels and model runtime, attention data parallelism with balanced context parallelism, MoE expert parallelism with expert load balancing (EPLB), and an asynchronous “zero-overhead” batch scheduler that overlaps CPU scheduling with GPU work.
• Speculative decoding (DSpark): a draft model proposes up to five tokens, and a confidence head decides which of them are worth verifying. DeepSeek says this cuts verified draft tokens per step from 5.00 to roughly 3.2–4.4 with almost no change in accepted length, for +4–7% throughput at its production operating point (about 60 concurrent sequences per GPU rank).
• SWA bounded replay: on a cache hit, the engine recomputes only the last 128 tokens of sliding-window attention state instead of rebuilding it from scratch. DeepSeek says the decoder version nearly halves total prefill compute for V4.1 Flash.
• Determinism options: batch-invariant inference (identical results regardless of what else is in the batch) and deterministic sampling with a fixed seed. Both are NVIDIA-only.
• Structured outputs: regex, JSON schema, or EBNF grammar constraints during generation.
• Kernels: builds on DeepSeek’s earlier open-source libraries (FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileKernels) plus custom TileLang kernels. DeepSeek says its internal kernel versions are “broadly the same”, with end-to-end performance within 5%.
• Lineage: DeepSeek says DeepInfer was originally built primarily on vLLM, with ideas from SGLang and TensorRT-LLM.

deepseek-recipe is the other half. It is a set of Rust libraries with Python bindings (pip install deepseek-recipe) that convert Chat Completions, Responses, and Anthropic-style Messages requests into DeepSeek V4/V4.1 prompts and token IDs, then parse model output back into each API’s streaming format. DeepSeek says its logic matches the official API, so a self-hosted deployment can behave the same way. It also blocks a quiet but real attack: strings like <｜User｜> typed into user input or tool results are encoded as plain text, so they cannot inject control tokens.

What it takes to run

• Hardware: NVIDIA SM100 GPUs (the Blackwell data-center generation) or Huawei Ascend 950 NPUs. Hopper-class cards such as H100 and H200 are not on the supported list.
• Software: Linux, GCC 13+, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). You build from source with ./build.sh.
• Model: the default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017 from Hugging Face. No other models are supported yet. DeepSeek says inference code for future models will land in the same repository.
• Scale: DeepSeek’s own guidance is that a minimal 1 prefill + 1 decode deployment works, but the smallest efficient unit is about 64 GPUs for prefill plus 64 for decode, because decode runs best with wide expert parallelism (EP64 or more).

That is the honest boundary. The repository is a real, buildable engine, but it is sized for teams that already run GPU clusters. Solo developers and small teams will get more from the ideas than from the code.

The numbers DeepSeek published

The throughput figures are read from DeepSeek’s published Pareto charts. One detail matters: the decode chart assumes a fixed 75% per-token draft acceptance rate rather than measured acceptance on real traffic, so treat it as a modeled ceiling for that setup. DeepSeek ships the sweep scripts to reproduce the charts, which is more than most vendors offer, but nobody outside DeepSeek has published a reproduction yet.

The $38 figure also needs care. It is revenue, not profit, and DeepSeek does not say which accelerator it refers to. A Hacker News commenter compared it with rental prices of a few dollars per H100-hour, but that comparison mixes different chips and leaves out power, storage, networking, and staff.

The repository also lists reference accuracy for V4.1 Flash under DeepInfer’s defaults: 99.9% on AIME 2026, 93.4% on GPQA Diamond, and 42.6% on HLE. DeepSeek frames these as a sanity check that your deployment is configured correctly, not as new model claims.

What is not in the box

DeepSeek says plainly that its KV cache management and load balancing are “tightly coupled to our internal infrastructure” and are not open-sourced yet. That leaves out three things a production service needs:

1. A KV cache system. DeepSeek’s runs on its 3FS distributed file system, with roughly 14.4 TB of logical cache per online GPU. Long cache retention is a big reason its cached-input prices can be so low.
2. Vision encoder / prefill / decode disaggregation. DeepSeek says most of the engine functionality is already there. The orchestration is not.
3. Load balancing across data-parallel replicas for prefill and decode.

The deployment Q&A describes how each of these works in detail, from the four context-length tiers its prefill balancer uses to how its decode balancer weighs request count against KV length. So the design is public, but you still have to write the code yourself. One Hacker News commenter summed it up: enough “to stir the pot but not enough to cook your own from scratch.”

What it means for builders

1. If you use DeepSeek through an API, expect hosting prices to get more competitive. OpenRouter already lists about 20 providers serving V4.1 Flash 1017, several of them priced below DeepSeek’s own listing there. Providers who adopt DeepSeek’s kernels, speculative decoding, and caching approach can push costs down further. You benefit without touching the repository.
2. If you run inference infrastructure, read the Q&A before the code. The practical lessons are in DeepSeek’s operating choices: wide expert parallelism for decode, a dedicated KV storage tier instead of prefix-affinity routing, confidence-scheduled speculative decoding, and long cache TTLs tuned by traffic type.
3. Compare it with your current engine on your own hardware. vLLM and SGLang already serve DeepSeek models on far more hardware. DeepInfer is specialized for one model on two chip families. Whether it beats a tuned vLLM or SGLang setup on SM100 is still an open question until someone runs the comparison.
4. Use deepseek-recipe even if you skip DeepInfer. If you self-host DeepSeek weights behind any engine, a tokenizer and parser that match the official API’s behavior (and block special-token injection) is useful on its own.
5. Note the Ascend support. First-class Huawei Ascend 950 support in a production engine is a real signal for teams and governments looking at non-NVIDIA hardware, and for anyone tracking how export controls shape the AI stack.

What to watch

• Independent benchmarks of DeepInfer against vLLM, SGLang, and TensorRT-LLM on SM100.
• Whether DeepSeek open-sources the KV cache and load-balancing components it held back.
• Support for more models, and whether vLLM and SGLang pick up DSpark and SWA bounded replay.
• Price moves from third-party DeepSeek hosts over the next few weeks.
• Policy friction. A US Department of Homeland Security order removing DeepSeek products from federal civilian agencies became public the same week. Self-hosting open code and weights does not automatically put a team outside that kind of rule, so government-facing teams should check before deploying.

Key takeaways

• DeepSeek open-sourced DeepInfer (Apache 2.0), the engine behind its V4.1 Flash service, plus deepseek-recipe (MIT) for API and tokenizer handling.
• It supports one model, DeepSeek V4.1 Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
• DeepSeek-reported: 155T tokens a day from one cluster, about $38 revenue per GPU-hour off-peak, and up to about 26K decode tokens/s per GPU on 64 GPUs.
• The KV cache system, stage disaggregation, and load balancing are not included. DeepSeek’s efficient minimum is about 128 GPUs.
• The biggest value for most builders is indirect: cheaper third-party hosting and a rare, detailed public design for frontier-scale inference.
• No independent benchmarks yet, and 9to6AI has not run it.

Sources

• DeepSeek: DeepInfer repository and README
• DeepSeek: V4.1 Flash inference deployment Q&A
• DeepSeek: DeepInfer evaluation reference results
• DeepSeek: deepseek-recipe repository
• DeepSeek announcement on X
• DeepSeek: inference system blog post (Zhihu, Chinese)
• Quartz: DeepSeek open-sources DeepInfer
• OpenRouter: DeepSeek V4.1 Flash 1017 providers
• Hacker News discussion