Radar / AI infrastructure / FreeToken
FreeToken: datacenter-scale MoE serving on desktop GPUs
An edge-native mixture-of-experts serving engine from UC Berkeley researchers that runs 290B+ parameter MoE models on consumer gaming GPUs, with bandwidth-adaptive CPU-GPU co-execution and semantic-aware caching.
Why it matters
It breaks the assumption that giant MoE models need datacenter hardware, streaming experts from disk and CPU so a desktop can serve models far larger than its GPU memory.
What you could build with it
A small AI consultancy could run a 290B-class MoE model on an office gaming PC with FreeToken and offer clients private, on-premise document analysis without paying datacenter GPU rental or sending data to a hosted API.
Does it hold up?
Early but concrete: the engine is public with 16K stars and a clear desktop-serving claim, but real-world throughput on consumer GPUs outside the authors' tests is not yet independently documented.
Built with FreeToken
- GitHub trending, day 264: FreeToken's desktop serving debutarticle · Notes FreeToken at 16,317 stars and +653 on the day, alongside other local-inference projects like colibri and airllm.
- GitHub Trending: FreeToken among the execution-infrastructure wavearticle · Places FreeToken in the local-inference layer of the agent ecosystem, from UC Berkeley authors including Matei Zaharia and Ion Stoica.
Learn more
First spotted on github: source.
More AI infrastructure
colibri: run 744B-parameter MoE models on consumer hardwarePure-C inference engine with zero dependencies that streams MoE experts from NVMe disk, keeping hot experts on GPU…infra · JEV 0.78claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74TorchTPU: Google's native PyTorch stack for TPUs, now open sourceAnnounced at PyTorch Conference 2026 on October 20: Google open-sourced TorchTPU, its native PyTorch backend for Cloud…infra · JEV 0.66Mesh-LLM: pool GPUs across machines behind one OpenAI-compatible endpointRust project that splits large models across machines into layer stages, fetches model shards by package, and serves…infra · JEV 0.64DeepInfer: DeepSeek's open-source production inference engineDeepSeek open-sourced DeepInfer, the high-performance inference framework powering its production services (155…infra · JEV 0.6Message Threads: Anthropic's public beta that stops agents resending the whole conversation every turnMessage Threads lets Messages API apps send only new messages after an initial full-conversation request, cutting…infra · JEV 0.57
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.