Radar / AI infrastructure / vLLM v0.34.0
vLLM v0.34.0: extensible KV cache by default, Model Runner V2 only, and 10 new models

In today's Spotlight · Picked by 4 of 4 modelsHelps services run more AI models and answer requests faster.Like upgrading the engine behind a busy delivery service.
The open inference engine's latest release spans 1,061 commits from 412 contributors, adds 10 models including Mistral Large 4, makes extensible KV cache sizing the default, and removes Model Runner V1.
Why it matters
The serving layer nearly every open-weight deployment runs on just shipped its biggest release in months, with 1.95x decode gains on Blackwell and a breaking change (the V1 runner is gone) every operator must plan around.
What you could build with it
A startup serving open-weight models could upgrade a staging cluster to v0.34.0 first, measure the extensible KV cache effect on its longest-context workload, and only then roll it out, using the larger cache to serve more concurrent long-document sessions on the same GPUs.
Does it hold up?
Battle-tested path: vLLM is the default serving engine for open models, but this is a breaking release, so operators should stage the upgrade since Model Runner V1 setups fail at startup and ROCm artifacts are 10.0-only.
Built with vLLM v0.34.0
- MiniMax-M3 on H200: KV cache grows from 433k to 672k tokens with extensible sizingkblip · The release notes measured extensible KV cache sizing growing the cache substantially on H200 hardware, at a 6-18% time-per-output-token cost.
- 1.95x decode on Step5 across 4x GB200 Blackwell GPUskblip · Blackwell decode speedups reported in the release, alongside 1.6x throughput on GLM-5.3-Flash and faster time-to-first-token on long contexts.
Learn more
First spotted on github: source.
More AI infrastructure
llama.cpp v1.0.0: the local-AI engine's first stable releasellama.cpp and ggml hit v1.0.0, promising semantic versioning so breaking C API changes only arrive with major bumps.infra · JEV 0.84colibri: run 744B-parameter MoE models on consumer hardwarePure-C inference engine with zero dependencies that streams MoE experts from NVMe disk, keeping hot experts on GPU…infra · JEV 0.78claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74FreeToken: datacenter-scale MoE serving on desktop GPUsAn edge-native mixture-of-experts serving engine from UC Berkeley researchers that runs 290B+ parameter MoE models on…infra · JEV 0.74GBrain: Garry Tan's self-wiring memory layer for AI agentsY Combinator CEO Garry Tan's open-source, markdown-first knowledge layer that gives OpenClaw and Hermes agents…infra · JEV 0.71AWS Emulator: a local copy of AWS, written entirely by AI agentsAWS's free, open-source local emulator covering 200+ services from 438 API models, written in Rust, running offline on…infra · JEV 0.67
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.