Radar / AI infrastructure / TrnRT
TrnRT: an open LLM inference engine for AWS Trainium
Why it matters
The first open engine that makes Trainium a first-class LLM serving target, with per-layer NKI megakernels and 1M-token context - an alternative to GPU-only serving stacks for teams already on AWS Trainium.
What you could build with it
A startup already running Trainium instances on AWS could serve GLM-5.3-Flash through TrnRT's OpenAI-compatible API, replacing a GPU fleet for that workload while keeping existing clients unchanged.
Does it hold up?
Too early to judge - first release (0.1.0) supports only GLM-5.3-Flash on one Trainium2 instance type, with no independent benchmarks or production reports yet.
Built with TrnRT
- trnrt 0.1.0 on PyPIpypi · First release (0.1.0, Nov 8): quickstart, benchmarks, and the GLM-5.3-Flash serving recipe on Trainium2.
- RunanywhereAI/trnrt - 1.6k starsgitstars · Tracking page: 1,622 stars as of Nov 18, repo created Oct 30.
- RunanywhereAI/trnrt repositorygitofanton · Repo stats: created Oct 30, last commit Nov 9, +1,321 stars since tracking began.
Learn more
First spotted on github: source.
More AI infrastructure
vLLM v0.34.0: extensible KV cache by default, Model Runner V2 only, and 10 new modelsThe open inference engine's latest release spans 1,061 commits from 412 contributors, adds 10 models including Mistral…infra · JEV 0.86llama.cpp v1.0.0: the local-AI engine's first stable releasellama.cpp and ggml hit v1.0.0, promising semantic versioning so breaking C API changes only arrive with major bumps.infra · JEV 0.84Graphify: turn any codebase into a knowledge graph your agent queriesA CLI plus agent skill that parses code with tree-sitter into a queryable knowledge graph of functions, classes…infra · JEV 0.77FreeLLMAPI: one free endpoint across 87 LLM providersA self-hosted router that puts 87 free LLM providers and 1,313 free model endpoints behind one OpenAI-compatible /v1…infra · JEV 0.73DeepSeek App API: the whole consumer app, now as developer APIsDeepSeek opened its app's capabilities through its API: web search, file parsing, URL reading, safety checks, voice…infra · JEV 0.67InferSSD: run 800B-class MoE models from an SSD on consumer hardwareWiCi's open-source inference engine streams MoE expert weights from NVMe so models larger than RAM or VRAM can run…infra · JEV 0.64
Get the week's best AI launches, plus 3 ideas worth building
One email every Saturday. Ranked by traction, not hype. Free.