Radar / Ideas / ScaffoldDiff: the regression gate that…

ScaffoldDiff: the regression gate that blames the right prompt, not the model

daily ideamoderateJEV confidence 0.59JEV monetization 0.552026-10-18

OutcomeEvery agent prompt, tool, skill, or guardrail change ships with measured proof of zero silent regressions — and the exact scaffold component responsible when one appears.

The problem

Teams running AI agents in production fear prompt changes: one line of prompt breaks behavior with no stack trace ('it used to work yesterday'). Existing eval tools flag that something regressed, but none can tell you WHICH scaffold component — the prompt, the guardrail, the skill, the tool schema — caused it, and none propose a tested fix.

The idea

A CI service that versions all eight scaffold components around a frozen model (prompts, guardrails, memory rules, tools, subagents, skills, scripts, plugins). On every change it replays a golden behavioral suite, attributes any regression to the exact component via ablation diffing, then proposes a refinement in the Harness² Propose–Probe–Compose style — gated by a typed decision-model verdict before the fix reaches the developer. The ship report reads: 'your guardrail edit, not the model swap, broke refunds; here is the tested rewrite.'

Why now

Google's Harness² (Oct 2026) proved the next frontier is scaffolding, not weights: recursive refinement of the 8 scaffold files with no verifiers and no retraining. Meanwhile decision models (Liquid d1-ultra at 65.2 on the AA Decision Index, StartLux v1.1 at 69.1 on the HF Decision Index) made cheap, calibrated pass/fail verdicts available. Detection tooling (EvalView, agentevals, AgentV) exists — diagnosis and verified auto-fix do not.

What it combines

Harness²'s component-level optimization loop (entry harness-google-s-recursive-agent-harnessing-framework) crossed with Jev's typed judgment layer (entry jev-1-14-0-typesafe-s-judgment-layer-update): the optimization loop proposes fixes, the decision layer certifies them as real improvements rather than benchmark overfits.

MVP

Weekend MVP: a CLI that (1) snapshots an agent's scaffold files into a versioned manifest, (2) runs a user-supplied golden input set before/after a change, (3) attributes regressions via single-component ablation, (4) outputs a markdown report naming the culprit file. Deliberately skip: the auto-fix proposal loop, the decision-model verdict gate, CI integration, and any hosted dashboard.

How it makes money

Budget owner: Head of AI Platform / ML Platform Lead — already pays for observability. Pricing anchor: Braintrust Pro $249/mo, Langfuse from $29/mo, LangSmith $39/seat/mo (dev.to and Braintrust's own comparison). Willingness-to-pay evidence: teams publicly document the pain — 'One Line Of Prompt Broke's Coding Brain' production-hardening guide, Evalvista's agent-regression-testing guide, and multiple dev.to postmortems on silent prompt regressions in production. Replacement budget: they already pay for detection; this adds the diagnosis-and-fix layer those tools lack.

The moat

Langfuse, Braintrust, and LangSmith are trace-centric and framework-tied: their eval loops replay against their own traces and assume scaffolding is static. ScaffoldDiff needs the inverse dataset — scaffold version-history crossed with failure-outcome pairs across frameworks — which none of them collect, and rebuilding their architecture around scaffold-as-code would cannibalize their traces-as-truth abstraction. Telling customers 'your traces aren't enough' would also undercut their own product story.

Distribution

CI-native: a GitHub Action that platform teams drop into agent repos. Sell to ML/AI platform leads at companies running agents in production; partner with agent frameworks (OpenCode, Codex executors) since Harness²-style optimization is executor-agnostic.

Why it wins

EvalView, agentevals, AgentV, Langfuse, and Braintrust all detect regressions via traces or golden replays; none attribute a regression to the exact scaffold component, and none close the loop with a verified fix proposal. This moves the category from 'something broke' to 'this file broke it — here is the tested patch.'

Risks

Golden suites are the bottleneck — teams without recorded production traffic cannot regression-test. The MVP de-risks this with a record mode that builds the golden set from a week of production traffic first. Second risk: ablation cost (N replays per change); de-risk with component-level result caching.

Build it with

Harness²: Google's recursive agent-harnessing frameworkPropose–Probe–Compose refinement loop that generates candidate scaffold fixes without verifiers.
Jev 1.14.0 — TypeSafe's judgment layer updateTyped verdicts as the regression pass/fail gate certifying each proposed fix.
Liquid AI d1-ultra: SOTA decision model with zero output tokensCalibrated decision scores for threshold-routing uncertain regressions to human review.

Repo to start from

scaffold-diff — git-blame for AI agent scaffolding: version, replay, attribute, and fix regressions across prompts, guardrails, skills, and tools.

Evidence

Get the week's best AI launches, plus 3 ideas worth building

One email every Saturday. Ranked by traction, not hype. Free.