TraceEvolve: the self-improving loop for coding-agent fleets

The problem

Teams running coding agents collect enormous trace volumes and do nothing with them: one practitioner reports recording over 100,000 traces a day while error patterns repeat invisibly across runs. Prompt tuning is still manual archaeology — an engineer reads traces, guesses at fixes, and hopes. Existing observability tells teams what happened, but the improvement loop back into the agent's prompts, skills, and routing stays a human job.

The idea

Why now

Three unlocks converged in the last month: LiteLLM Lens puts trace-analyzing agents inside the gateway where all model traffic already flows; Reflexio demonstrated agents that learn from corrections with no retraining; and Microsoft open-sourced 301,000 real Copilot coding-agent traces — a seed corpus of genuine failure patterns that lets the loop bootstrap a failure-to-fix playbook library on day one instead of starting cold.

What it combines

LiteLLM Lens (agent-driven trace analysis inside the gateway) + Reflexio (self-improvement from corrections, no retraining) + Microsoft's 301k Copilot trace corpus (real failure patterns). Trace analysis alone is read-only insight; correction learning alone has no pattern library to start from; the trace corpus alone is inert data. Together they form a closed loop — observe at the gateway, learn from corrections, bootstrap from real-world failures — that compounds fleet quality weekly.

MVP

Weekend scope: ingest LiteLLM-gateway traces for one coding agent, cluster failures into patterns, attach engineer corrections from a linked repo, evolve one prompt/skill file, and show a before/after pass rate on a small labeled eval set. Deliberately skip: multi-model support, auto-drafting code PRs, long-term memory, and the SaaS dashboard — a local HTML report is enough.

Distribution

B2B2C: embed as the improvement engine inside open agent harnesses and control planes (OpenRig, OpenClaw Enterprise), and sell direct to engineering orgs running coding-agent fleets. Open-core model — the trace-to-evolution loop is open source and self-hosted; the fleet eval dashboard and failure-pattern library are paid SaaS.

Why it wins

LangSmith Engine just launched auto-debugging that drafts fix PRs, but it lives inside LangChain's tracing ecosystem. Braintrust, Humanloop, and Arize focus on evaluation and insight, not automatic evolution. TraceEvolve is the neutral, self-hosted layer for the other 90%: teams running open or local coding-agent harnesses (OpenRig, OpenClaw) across any model provider, with the loop closed from trace to evolved harness config.

Risks

Auto-evolved prompts can silently regress behavior the eval set doesn't cover. The MVP de-risks this by making evolution proposal-only: nothing ships until the gated eval suite passes, and a human approves the first changes while the gate's precision is being measured.

Build it with

• LiteLLM Lens — AI agents that analyze agent traces inside the gatewayTrace analysis at the gateway layer, where all model traffic already flows — the observation point for the loop.
• Reflexio: self-improving agents that learn from corrections, no retrainingThe correction-driven learning mechanism that evolves agent behavior without retraining.
• Microsoft releases 301,000 Copilot coding-agent tracesSeed corpus of real coding-agent failure patterns to bootstrap the failure-to-fix playbook library on day one.
• OpenRig: open-source local control plane that runs Claude Code and Codex as one managed teamA concrete open harness to embed the improvement loop in first — the B2B2C distribution wedge.

Repo to start from

traceevolve — open trace-to-correction-to-prompt-evolution loop: a LiteLLM gateway plugin, failure clustering, correction ingestion from git history, and a gated eval runner that approves or rejects each evolved config.

Evidence

• LangSmith Engine closes the agent debugging loop automatically — but multi-model enterprises still need a neutral layer — VentureBeat
• I thought plugging in LangSmith would solve agentic AI monitoring — DEV
• AI engineering bazaar: observability module — '100,000 traces every single day, and what are they doing with those traces? Literally nothing.'