Radar / AI infrastructure / DeepInfer
DeepInfer: DeepSeek's open-source production inference engine
DeepSeek open-sourced DeepInfer, the high-performance inference framework powering its production services (155 trillion tokens/day per cluster, 96% input cache hit rate), under Apache 2.0 on October 20, 2026.
Why it matters
Not a paper — the actual production serving stack: FlashMLA, DeepGEMM, DeepEP and TileKernels kernels, DSpark confidence-scheduled speculative decoding, batch-invariant inference, deterministic sampling, and dual Nvidia SM100 + Huawei Ascend 950 support. A reversal of DeepSeek's April-2025 decision not to release its engine.
What you could build with it
A regional GPU cloud or neocloud could offer managed DeepSeek-V4.1-Flash inference built on DeepInfer, using its async batch scheduler and expert-parallel MoE layout to match DeepSeek's own serving cost on SM100 clusters and undercut managed inference providers on open-weight MoE models — the release hands any operator with datacenter GPUs the lab's cost advantage.
Does it hold up?
Very strong early signal: 13k+ GitHub stars and multiple independent hands-on writeups within 24 hours. Real self-hosting is datacenter-grade (a production-shaped deployment needs ~128 top-end accelerators per early analysis) and currently serves only DeepSeek-V4.1-Flash.
Built with DeepInfer
- DeepSeek open-sources DeepInfer, the engine behind its cheap AImadrobot.blog · Writeup of the October 20 release: the April-2025 reversal, dual SM100/Ascend-950 hardware targets, and Apache 2.0 licensing that lets anyone self-host commercially.
- DeepInfer: DeepSeek's Free Engine Needs 128 GPUs to Runyoutube.com · Video breakdown of what shipped versus what stayed on DeepSeek's servers (load balancers kept home), build/run commands, and deployment economics — production shape needs ~128 top-end accelerators.
- How to Speed Up DeepSeek with DeepInfer: installation and inference setupai-release.org · Hands-on setup guide published within a day of release: host prerequisites, cloning, building with CUDA 13, and running inference against DeepSeek-V4.1-Flash weights.
- deepseek-ai/DeepInfergithub · 13,335 ★ · The engine itself: vLLM-derived async scheduler, FlashMLA/DeepGEMM/DeepEP kernels, DSpark speculative decoding, batch-invariant mode, regex/JSON-schema/EBNF constrained generation.
- deepseek-ai/deepseek-recipegithub · 1,610 ★ · Companion package released alongside DeepInfer handling token encoding/decoding aligned with the official DeepSeek API for self-hosted deployments.
Learn more
First spotted on github: source.
More AI infrastructure
DeepGEMM: DeepSeek's open GPU BLAS kernel libraryDeepSeek's DeepGEMM, a clean and efficient open-source BLAS kernel library for NVIDIA GPUs (up to 1550 TFLOPS on…infra · JEV 0.77OpenViking — self-evolving context database for AI agentsByteDance Volcengine's open-source (AGPL-3.0) context database that unifies agent memory, knowledge RAG, and skills…infra · JEV 0.76Modular open-sources MAX inference server, Mojo stdlib and accelerator kernelsThe core of Modular's unified AI deployment stack is now open: the MAX inference server (OpenAI-compatible endpoints)…infra · JEV 0.76ai-memory - git-backed long-term memory shared across coding agentsRust server and CLI that captures agent sessions via lifecycle hooks, compiles them into a git-backed markdown wiki…infra · JEV 0.76OmniRouteFree MIT AI gateway: one endpoint, 290+ providers and 500+ models with quota-aware auto-fallback.infra · JEV 0.75Context Mode: context-window optimization for AI coding agentsMCP server for AI coding agents that sandboxes verbose tool output and persists session memory, cutting context use by…infra · JEV 0.74
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.