Radar / AI infrastructure / vLLM v0.32.0
vLLM v0.32.0
Adds POST /v1/systemone, a structured-decisions endpoint that answers choice, noul, and score questions over structured state directly inside the inference engine, plus a CORS default lockdown that stops any website from calling a local model server.
Why it matters
Decision routing moves into the inference server itself — agent routers skip the double-prefill tax of calling a separate judge model over HTTP and keep the KV cache resident.
What you could build with it
A startup building real-time voice agents could run its turn-taking classifier through the same vLLM server as the main model, getting a calibrated answer/stop/backchannel decision in-engine instead of a separate judge call — trimming per-turn latency on every conversational exchange.
Does it hold up?
Too early to judge — the release landed Oct 18 and production reports don't exist yet, though the feature is verified in the official release notes and LiteLLM's docs already describe how gateways translate to it.
Built with vLLM v0.32.0
- Why vLLM 0.32 Baked Decision Routing Into the EngineReid Marlow · Explains the design rationale: moving routing decisions inside the engine avoids paying the prefill tax twice and stops the KV cache from being discarded.
- vLLM 0.32 stops letting any website call your model serverDEV Community · Documents the release's other headline change: the --allowed-origins default flips from wildcard to deny, closing a long-standing local-server exposure.
- System One: Typed Decisions — ChangelogLLM Gateway · Documents the System One typed-decisions API (noul/choice/score) that vLLM's new endpoint implements, with probability-attached answers that code can branch on.
Learn more
First spotted on github: source.
More AI infrastructure
DeepGEMM: DeepSeek's open GPU BLAS kernel libraryDeepSeek's DeepGEMM, a clean and efficient open-source BLAS kernel library for NVIDIA GPUs (up to 1550 TFLOPS on…infra · JEV 0.77OpenViking — self-evolving context database for AI agentsByteDance Volcengine's open-source (AGPL-3.0) context database that unifies agent memory, knowledge RAG, and skills…infra · JEV 0.76Modular open-sources MAX inference server, Mojo stdlib and accelerator kernelsThe core of Modular's unified AI deployment stack is now open: the MAX inference server (OpenAI-compatible endpoints)…infra · JEV 0.76ai-memory - git-backed long-term memory shared across coding agentsRust server and CLI that captures agent sessions via lifecycle hooks, compiles them into a git-backed markdown wiki…infra · JEV 0.76OmniRouteFree MIT AI gateway: one endpoint, 290+ providers and 500+ models with quota-aware auto-fallback.infra · JEV 0.75Context Mode: context-window optimization for AI coding agentsMCP server for AI coding agents that sandboxes verbose tool output and persists session memory, cutting context use by…infra · JEV 0.74
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.