Radar / AI infrastructure / Together Compiler
Together Compiler: AI agents that continuously retune inference
Together AI launched Together Compiler, a system in which teams of agents continually rewrite and measure GPU code for the inference engines it serves, so new open models get faster overnight instead of after weeks of hand tuning.
Why it matters
Persistent engines hot-reload candidate forward passes instead of restarting, cutting time-to-serving from 467 seconds to 9 seconds in the company's comparison and delivering up to 2.6x throughput on frontier open models in under 24 hours.
What you could build with it
An indie developer running a small GPU fleet could build a nightly auto-tuner that re-benchmarks each hosted model against the day's traffic shape and swaps in the fastest kernel configuration, cutting inference bills without any manual performance work.
Does it hold up?
Usable only through Together AI itself for now: it already runs across many of the company's serverless endpoints, but outside access is a limited beta, so independent teams cannot yet run it on their own fleets.
Built with Together Compiler
- Together AI launches Together Compiler for continual optimizationarticle · Launch announcement with the headline numbers: 252x more experiments than existing agentic optimization systems and 1.5x/1.4x peak throughput gains on Kimi K3 and GLM 5.3.
- Together AI brings continuous compilation to production inferencearticle · Independent coverage explaining the persistent-engine design, the verification tests for KV-cache contamination, and the limited-beta availability.
Learn more
First spotted on article: source.
More AI infrastructure
colibri: run 744B-parameter MoE models on consumer hardwarePure-C inference engine with zero dependencies that streams MoE experts from NVMe disk, keeping hot experts on GPU…infra · JEV 0.78claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74FreeToken: datacenter-scale MoE serving on desktop GPUsAn edge-native mixture-of-experts serving engine from UC Berkeley researchers that runs 290B+ parameter MoE models on…infra · JEV 0.74TorchTPU: Google's native PyTorch stack for TPUs, now open sourceAnnounced at PyTorch Conference 2026 on October 20: Google open-sourced TorchTPU, its native PyTorch backend for Cloud…infra · JEV 0.66Mesh-LLM: pool GPUs across machines behind one OpenAI-compatible endpointRust project that splits large models across machines into layer stages, fetches model shards by package, and serves…infra · JEV 0.64DeepInfer: DeepSeek's open-source production inference engineDeepSeek open-sourced DeepInfer, the high-performance inference framework powering its production services (155…infra · JEV 0.6
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.