Radar / Ideas / CampaignHoldout: test whether an…

CampaignHoldout: test whether an SEO-led page and promo beat the current campaign

weekly debate · 2026-W44moderateJEV confidence 0.54JEV monetization –2026-10-30
OutcomeIncrease qualified landing-page conversion rate in an agency-run A/B test, measured against a concurrent holdout using customer-provided counts.

The problem

claude-seo identifies page issues, and Video Shotcraft makes promos from product pages. Agencies need a disciplined way to test whether an approved page change and matching promo perform better together.

The idea

Turn one audit finding into one approved page variant, then render a promo from that exact variant. Give the agency a test plan for splitting its own campaign traffic between the existing page and promo and the new pair; import counts to compare results.

Why now

claude-seo can produce an audit, while Video Shotcraft can make an editable promo using product-page screenshots and copy. The pair supports a consistent, testable campaign variant.

What it combines

The SEO finding determines the page change; the changed page becomes the promo’s source. This creates a matched page-and-video variant rather than two unrelated deliverables.

How to build it

  1. Define the test. Record the qualified-conversion definition, existing campaign and agency-controlled traffic split. (1 day)
  2. Create the variant. Run an authorized audit and obtain client approval for one finding-led page change. (1 day)
  3. Render the matched promo. Produce a video from the approved variant’s page and copy. (1 day)
  4. Compare arms. Import concurrent traffic and conversion counts; report both rates and any inconclusive result. (1 day)

Who benefits

Agencies
Can test a coherent campaign variant rather than deliver disconnected assets.
Business clients
Can approve the change and inspect a holdout comparison.

How it makes money

Model
Agency subscription
Pricing
$199 per agency per month for the pilot
Wedge
One agency’s next product-page campaign
Channels
Direct outreach to small marketing agencies · Agency invitations to existing clients

Distribution

Marketing agencies pay and invite business clients to approve variants. The wedge is an agency’s next product-page campaign; its existing client roster provides B2B2C distribution.

Why it wins

Instead of delivering an audit and video separately, CampaignHoldout preserves the link from finding to approved variant to concurrent test result. It reports the pair’s performance, not the independent effect of either asset.

Risks

A small or poorly split test may be inconclusive. Require the agency to document allocation, dates and conversion definitions; report raw rates and mark insufficient data as inconclusive.

Strongest objection left standing

Even a well-run holdout tests the page-and-video bundle, not whether the SEO finding or video deserves credit.

Build it with

Repo to start from

campaignholdout - matched page-and-video variants with holdout reporting

Evidence

Debated by openai/gpt-6-sol, anthropic/claude-sonnet-5.5, qwen/qwen3.8-max-0902. Judged by JEV.

Get the week's best AI launches, plus 3 ideas worth building

One email every Saturday. Ranked by traction, not hype. Free.