Shipwright / collaborative A/B testing

Compare two directions. Decide together before merge.

Shipwright puts Variant A and Variant B in one decision room. It captures both in the browser, gives AI judges and human reviewers the same evidence, keeps dissent visible, and promotes a winner only when the gates pass.

Open Shipwright See how the decision works

Example decision

Which direction should reach the agent?

Pre-merge
A Selected

Experiment cockpit

Variants, evidence, votes, and dissent share one surface.

AI judge AI judge Human vote
B Runner-up

Fast winner board

The recommendation lands first; deeper evidence follows.

AI dissent kept
Browser capture Model panel Human review Dissent
Decision packet Variant A → bounded worker handoff

The product loop

The decision happens before the code lands.

Most review starts after someone has already committed to a direction. Shipwright moves product judgment earlier, while a competing version is still cheap to compare and human feedback can still change the work.

01

Frame two variants

Define the question, hypotheses, audience, expected signal, and Variant A/B allocation.

02

Capture the real surfaces

Render each direction and collect Playwright screenshots and interface signals as evidence.

03

Judge together

AI models and human reviewers choose, explain why, and leave minority views visible.

04

Promote—or hold

Only a winner with enough evidence becomes a worker prompt. Missing proof stops promotion.

AI + human collaboration

One evidence packet. Different kinds of judgment.

Model judges return a selection, confidence, rationale, and the evidence they used. Human review adds product context, must-fixes, acceptance, or dissent. Shipwright keeps both instead of flattening the decision into one unexplained score.

AI
Reasoning judge

The comparison surface exposes the decision trail without requiring raw logs.

Vote A
AI
Visual judge

Variant B is faster to scan. Keep that advantage in the implementation checklist.

Vote B
H
Human reviewer

Choose A, but retain the runner-up’s lighter first impression.

Vote A
1 2

Surface Grill / human review

Point at the problem, not just the page.

Surface Grill anchors must-fixes, accepted feedback, and dissent to normalized coordinates on a captured variant. Each review creates a digest-bound revision packet for the next pass; the review surface itself never edits the source project.

Decision gates

A winner is not automatically ready to build.

Shipwright separates “this direction won” from “there is enough proof to promote it.” A missing screenshot or incomplete handoff keeps the result in needs-more-signal—even when the weighted score favors one variant.

Promotion checklist

Every required signal stays inspectable.

  1. 01Weighted decision thresholdRequired
  2. 02Judge alignment or labeled dissentRequired
  3. 03Human vote state, including pendingVisible
  4. 04Rendered Playwright evidenceRequired
  5. 05Verification commands and worker promptRequired

The pre-merge handoff

The agent gets a decision, not a vague request.

Once the gates pass, Shipwright turns the winning direction and the strongest dissent into a bounded work order. The next agent sees what to change, what not to lose, and how the result will be verified before merge.

Decision → implementation

Keep the why attached to the work.

The selected variant, runner-up risk, review notes, source boundary, acceptance criteria, and next verification command travel together. Review happens before commitment to the direction—and remains available when the patch comes back for approval.

Worker handoff Ready for build
Selected
Variant A and its implementation scope
Preserve
The runner-up’s strongest advantage and recorded dissent
Boundary
Source files allowed for the first edit
Accept when
The visual and behavioral criteria pass
Verify
The next exact command for the worker to run

Proof Story / after release

Turn the receipt into a story—without outrunning the proof.

Once the selected change is on main and its public checks pass, Shipwright builds a short release story from that exact receipt. Every scene stays bound to verified screenshots, source revision, checks, and public URL; stale or missing evidence blocks export.

Release → understanding

A release story that carries its receipts.

The first 15-second Proof Story uses a fresh capture from Shipwright’s production origin. It ships with the approved claims, source and deployment receipts, capture hash, poster, share copy, release note, and a machine-readable provenance manifest—without a model, music, telemetry, simulated interaction, or automatic posting.

Watch the first Proof Story Inspect its manifest
Proof packet Released
Feature source
3a4cadc · receipt-bound compiler
Published source
e80ed23 · browser-complete story route
Surface
Canonical Shipwright production origin
Capture
SHA-256 a80717f20591…
Story
SHA-256 9cf8c0b1683d…