Radar / AI infrastructure / InferSSD
InferSSD: run 800B-class MoE models from an SSD on consumer hardware
WiCi's open-source inference engine streams MoE expert weights from NVMe so models larger than RAM or VRAM can run locally — DeepSeek-V4.1-Flash (763B) and Muse-Spark-Open (816B) on a 48 GB MacBook or an RTX 5090, at a few tokens per second.
Why it matters
Turns the SSD into slow VRAM for MoE models: only the routed experts are read per token, so a 500 GB checkpoint runs on hardware that could never hold it.
What you could build with it
A researcher with a single gaming GPU could run a 700B-class open model locally to test prompts and evals that would otherwise need rented cloud GPUs, trading speed for zero hourly cost.
Does it hold up?
Works but slow: independent testing confirms the streaming runs, yet decode sits at 2-7 tok/s and plain llama.cpp was faster on the same models — a research curiosity for now, not a daily driver.
Built with InferSSD
- InferSSD: 800B LLMs on RTX 5090 SSD Pathexplainx.ai · Benchmark tables: DeepSeek on RTX 5090 at 164 tok/s prefill, 7.3 tok/s decode, with SSD-endurance caveats.
- Episode 12 — Smells Like Snake Oilinerlogic.com · Independent hands-on test: the streaming is real but speed estimates ran 2-3x high, and plain llama.cpp beat it on the tested models.
- InferSSD: run 800B models on a 48GB MacBook with an SSDyoutube.com · Short video walkthrough of the SSD expert-streaming idea and the install flow.
Learn more
First spotted on github: source.
More AI infrastructure
vLLM v0.34.0: extensible KV cache by default, Model Runner V2 only, and 10 new modelsThe open inference engine's latest release spans 1,061 commits from 412 contributors, adds 10 models including Mistral…infra · JEV 0.86llama.cpp v1.0.0: the local-AI engine's first stable releasellama.cpp and ggml hit v1.0.0, promising semantic versioning so breaking C API changes only arrive with major bumps.infra · JEV 0.84Graphify: turn any codebase into a knowledge graph your agent queriesA CLI plus agent skill that parses code with tree-sitter into a queryable knowledge graph of functions, classes…infra · JEV 0.77FreeLLMAPI: one free endpoint across 87 LLM providersA self-hosted router that puts 87 free LLM providers and 1,313 free model endpoints behind one OpenAI-compatible /v1…infra · JEV 0.73DeepSeek App API: the whole consumer app, now as developer APIsDeepSeek opened its app's capabilities through its API: web search, file parsing, URL reading, safety checks, voice…infra · JEV 0.67TrnRT: an open LLM inference engine for AWS Trainiuminfra · JEV 0.63
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.