Radar / AI infrastructure / billion-context
billion-context: month-long agent sessions in a 100K context window
A local proxy that sits between a coding agent and its model API, letting the model fold old messages into searchable notes so one session can run for months.
Why it matters
Model-driven incremental compression claims 5x fewer tokens and 95-97% prefix-cache hit rates, with folding the model controls itself — no hard truncation, and anything folded can be reopened on demand.
What you could build with it
An indie developer running a multi-day autonomous coding agent could route the agent through billion-context to keep one session alive across a large refactor, instead of losing context to auto-compaction mid-task.
Does it hold up?
Works and is genuinely used for long Claude Code sessions, but independent testing found no savings below roughly 100K tokens and a real cache-cold cost on some setups — worth it only for very long sessions on small windows.
Built with billion-context
- Billion-Context Setup for Claude Code: Long Chats Without Auto-Compactcodewithnishant.dev · Hands-on setup guide for Claude Code covering the proxy, the four compression tools, and the 95-97% cache-health rule of thumb.
- billion-context: a context-compression proxy, measured honestlyfreedzhost.cloud · Independent measurement across Claude Code, Kimi Code and OpenFox: no gain under about 100K tokens and a cold-cache cost on some setups, with the real payoff on very long sessions still unproven.
- Billion-Context: How 5x Token Compression Lets 100K-Window Agents Handle Month-Long Sessionsdev.to · Walkthrough of the compression pipeline and MCP-hook integration across 17 agent clients.
Learn more
First spotted on github: source.
More AI infrastructure
colibri: run 744B-parameter MoE models on consumer hardwarePure-C inference engine with zero dependencies that streams MoE experts from NVMe disk, keeping hot experts on GPU…infra · JEV 0.78claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74FreeToken: datacenter-scale MoE serving on desktop GPUsAn edge-native mixture-of-experts serving engine from UC Berkeley researchers that runs 290B+ parameter MoE models on…infra · JEV 0.74XLA Serving Engine: Google open-sources its C++ inference runtimeGoogle's general-purpose model export and inference runtime for the OpenXLA ecosystem, now open source under Apache 2.0.infra · JEV 0.67TorchTPU: Google's native PyTorch stack for TPUs, now open sourceAnnounced at PyTorch Conference 2026 on October 20: Google open-sourced TorchTPU, its native PyTorch backend for Cloud…infra · JEV 0.66Mesh-LLM: pool GPUs across machines behind one OpenAI-compatible endpointRust project that splits large models across machines into layer stages, fetches model shards by package, and serves…infra · JEV 0.64
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.