HarnessCert: the public scorecard for agent frameworks

The problem

Framework choice (LangGraph vs CrewAI vs Microsoft Agent Framework vs Strands) is the highest-stakes infra decision an AI team makes, and current guidance is vibes: vendor blog listicles ranking 'best frameworks' on marketing dimensions, and model benchmarks (τ-bench, GAIA, OSWorld) that score the underlying LLM rather than the harness. Enterprises are told to build their own golden datasets and evaluate candidates themselves — expensive, slow, and incomparable across vendors.

The idea

Why now

Hugging Face began hosting RL environments on the Hub on Oct 7 — environments as a shared, versioned primitive; ThinkingBox (Microsoft + Hugging Face, Oct 4) gives a vendor-neutral agent benchmark; ActiveSaddler (Oct 4) demonstrates automated curriculum learning for harness optimization; and trace-analysis tooling (LiteLLM Lens) makes automatic failure-mode attribution feasible.

What it combines

Four radar capabilities: hugging-face-hosts-rl-environments-on-the-hub (shared versioned environments), thinkingbox-agent-benchmark (neutral agent benchmark), activesaddler-curriculum-harness (self-hardening task curriculum), and litellm-lens (trace diagnostics). The mix is 'score AND improve': the scorecard does not just rank harnesses, it tells each framework exactly where it fails and generates the tasks to fix it — no single piece does that.

MVP

Build first: run 3 frameworks (LangGraph, CrewAI, Strands) on the τ-bench retail domain plus 2 HF RL environments and publish the first scorecard as a static site; get one framework vendor to pay for a re-run. Deliberately skip: curriculum task generation, trace diagnostics, and the formal certification program.

Distribution

B2B2C: sell to CI/devtool platforms and to Hugging Face as the hosted verification layer for the environments they now host; framework vendors pay for certification badges and monthly re-runs; enterprises get the public board free as top-of-funnel for paid private procurement evals.

Why it wins

τ-bench/τ2-bench, GAIA, and OSWorld evaluate agents and models on static task sets; none certify harnesses, none publish framework-vs-framework leaderboards, and none close the loop with curriculum-generated harder tasks or trace-based failure attribution. LangSmith-style observability is per-customer, not a public comparative rating.

Risks

Framework vendors will dispute methodology and claim the harness favors rivals. De-risk by making every run fully reproducible — pinned versions, public traces, an open rerun kit — the same dispute model the Decision Index's kit uses.

Build it with

• Hugging Face hosts RL environments on the HubShared, versioned RL environments on the Hub — the standard task substrate the scorecard runs on.
• ThinkingBox: Microsoft + Hugging Face agent benchmarkMicrosoft + Hugging Face vendor-neutral agent benchmark — the neutral scoring methodology.
• ActiveSaddler: automated curriculum learning for agent harness optimizationAutomated curriculum learning — generates harder follow-up tasks so frameworks can't overfit the scorecard.
• LiteLLM Lens — AI agents that analyze agent traces inside the gatewayAgent-trace analysis — attributes each framework's failure modes automatically for the scorecard's diagnostics section.

Repo to start from

harnesscert/harnesscert — reproducible framework certification harness: environment-pack runner, public scoreboard site, curriculum task generator.

Evidence

• Hugging Face hosts RL environments on the Hub
• tau2-bench — tool-agent-user benchmark (GitHub)
• 8 Best AI Agent Frameworks for Enterprise in 2026
• AI Agent Evaluation 2026: 7-Dimension Framework Fortune 500 CTOs Use