Radar / AI infrastructure / Mesh-LLM
Mesh-LLM: pool GPUs across machines behind one OpenAI-compatible endpoint
Rust project that splits large models across machines into layer stages, fetches model shards by package, and serves the result as a single OpenAI-compatible endpoint over QUIC-encrypted peer transport.
Why it matters
Lets a team run a model too large for any single box without building its own inference cluster, with single-machine fit preferred before any split.
What you could build with it
A team with two or three GPU machines could join them into a private mesh to serve an open-weights model that fits on none of them alone, keeping prompts inside its own network instead of paying a hosted API.
Does it hold up?
Still maturing: the README itself flags Windows distribution bugs on its front page and gives no end-to-end cross-machine split latency, so it suits experimentation on private meshes before production serving.
Built with Mesh-LLM
- Open Source Radar, 2026-10-31: Mesh-LLM among GitHub trending agent infrastructurearticle · Independent README-level review: shard-by-package fetching, single-machine-first scheduling, QUIC transport, and the README's own Windows caveats.
Learn more
First spotted on github: source.
More AI infrastructure
claude-mem: persistent memory plugin that lets coding agents pick up where the last session left offPlugin that hooks five Claude Code lifecycle events, compresses session activity into a local SQLite + Chroma store…infra · JEV 0.74TorchTPU: Google's native PyTorch stack for TPUs, now open sourceAnnounced at PyTorch Conference 2026 on October 20: Google open-sourced TorchTPU, its native PyTorch backend for Cloud…infra · JEV 0.66DeepInfer: DeepSeek's open-source production inference engineDeepSeek open-sourced DeepInfer, the high-performance inference framework powering its production services (155…infra · JEV 0.6Rival.io GraphX: run DeepSeek V4 Flash on eight consumer-grade NVIDIA L4 GPUsProprietary kernel-optimization and silicon-placement technology that ran DeepSeek V4 Flash on eight NVIDIA L4 GPUs…infra · JEV 0.35You.com Inference RouterOne OpenAI-compatible API for 12 open-weight models with a 99.9% uptime SLA, zero data retention, and built-in…infra · JEV 0.31EAGLE-4: one flexible speculative-decoding drafter for all batch sizes and hardwareEagle's newest speculative-decoding drafter adapts its draft tree to batch size, hardware, and workload, speeding up…infra · JEV 0.28
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.