Radar / AI infrastructure / Memorilla
Memorilla: latent semantic memory for LLMs
Stanford and Arc Institute memory system that compresses an entire document collection into 16 query-dependent memory tokens for a frozen LLM, beating full-context inference on FactKG while using 108x fewer tokens.
Why it matters
Memory as a modality: 16 learned tokens beat full context on five of eight benchmarks — the first memory system to outperform putting everything in context, not just compress it.
What you could build with it
A team building a long-running research assistant could give it Memorilla-style latent memory so the agent recalls months of past work in 16 tokens per query instead of re-reading its full history every turn.
Does it hold up?
Too early to judge: a research paper with strong benchmark numbers on a frozen Qwen3-8B; no public code release or production deployment has been shown yet.
Built with Memorilla
- Stanford researchers compress LLM memory into 16 tokens and beat full context on five benchmarksarticle · Coverage of the Nov 5 release, the 16-token design, and the caveat that results are so far on Qwen3-8B benchmarks.
- Memorilla: Latent Semantic Memory for LLMs — DAIR.AI Academyarticle · Paper explainer: the design, the 16-token mechanism, and benchmark results against Mem0 and MemoRAG.
Learn more
First spotted on article: source.
More AI infrastructure
colibri: run 744B-parameter MoE models on consumer hardwarePure-C inference engine with zero dependencies that streams MoE experts from NVMe disk, keeping hot experts on GPU…infra · JEV 0.78claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74FreeToken: datacenter-scale MoE serving on desktop GPUsAn edge-native mixture-of-experts serving engine from UC Berkeley researchers that runs 290B+ parameter MoE models on…infra · JEV 0.74XLA Serving Engine: Google open-sources its C++ inference runtimeGoogle's general-purpose model export and inference runtime for the OpenXLA ecosystem, now open source under Apache 2.0.infra · JEV 0.67TorchTPU: Google's native PyTorch stack for TPUs, now open sourceAnnounced at PyTorch Conference 2026 on October 20: Google open-sourced TorchTPU, its native PyTorch backend for Cloud…infra · JEV 0.66billion-context: month-long agent sessions in a 100K context windowA local proxy that sits between a coding agent and its model API, letting the model fold old messages into searchable…infra · JEV 0.65
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.