ship or theater

Is it shipping, or is it theater?

Saw an AI claim that sounds too good to be true?

Every day your feed says AI just replaced a team, passed the bar exam, or built an app over lunch. Some of it is real. A lot of it is a polished demo, a cherry-picked number, or plain hype. Paste the claim in. Ship or Theater asks the six questions a skeptical engineer would ask and gives you a straight answer, with reasons: it ships, it’s mixed, or it’s theater.

npx ship-or-theater "<claim>"

Open source. Use your Claude Code login or an API key.

Google DeepMindMixed

“our latest AlphaFold system achieves a median score of 92.4 GDT overall across all targets.”

65.8/ 100 reality score

Highest of the 8 gallery runs. 4.2 points below the Ships line at 70.

  1. Reproducibility 20% 68
  2. Evidence quality 20% 78
  3. Scope honesty 20% 55
  4. Production-readiness 15% 62
  5. Novelty 10% 88
  6. Hidden caveats 15% 50
Run Nov 7, 2026 with Claude Sonnet via Claude Code, ship-or-theater v1.0.0 · Full scorecard

How a claim gets judged

Six skeptics challenge the claim, a steelman makes its strongest honest case, and a judge weighs both sides. The result: a score from 0 to 100, or inconclusive when the evidence can’t support a number.

  1. Claim

    The text, plus optional context and a source link.

  2. Extractor

    Pulls out the specific assertions that can be tested.

  3. Six skeptics, in parallel

    One per dimension, each hunting for theater.

    • Reproducibility
    • Evidence quality
    • Scope honesty
    • Production-readiness
    • Novelty
    • Hidden caveats

    Steelman argues the strongest honest case for the claim.

  4. Judge

    Weighs skeptics against the steelman. Scores each dimension, or marks it inconclusive.

  5. Scorecard

    Weighted score, verdict, both sides of the argument. Terminal, Markdown or JSON.

If a skeptic fails, its dimension becomes inconclusive and drops out of the score. The run finishes; it never fills the gap with a guess.

The rubric is the point of view

Fixed, weighted and published. Each dimension scores 0 (pure theater) to 100 (real). The reality score is the weighted mean of the dimensions that had usable evidence.

Rubric dimensions and weights
DimensionThe questionWeight
ReproducibilityCould an independent party reproduce it, or is it a cherry-picked demo?20%
Evidence qualityReal benchmarks and data, or screenshots and vibes?20%
Scope honestyDoes it generalize, or is narrow work dressed up as general?20%
Production-readinessDemo or deployable: latency, cost, reliability, governance?15%
Hidden caveatsWhat is not said: failure modes, constraints, cost?15%
NoveltyGenuine new capability, or a wrapper and a rebrand?10%

Verdict bands

Ships
70 and up. Credible, checkable, close to deployable.
Mixed
40 to 69. Something real, wrapped in more than it earns.
Theater
Under 40. A demo dressed as a result.
Inconclusive
No dimension had usable evidence, so there is no score. The tool won’t invent one.

Set up in a minute

Node 20 or newer. npx ship-or-theater rubric works with no setup at all; scoring a claim needs a model, and your Claude Code login is enough. A full verdict makes nine model calls and takes a few minutes.

Score a claim. Context and a source link make the skeptics sharper.

# print the rubric, no setup needed
npx ship-or-theater rubric

# score a claim
npx ship-or-theater "Our 7B model beats GPT-4 on internal benchmarks." \
  --context "Launch post; no weights or eval suite released."

# pick the output: pretty (terminal default), markdown or json
npx ship-or-theater "<claim>" --format json --out scorecard.json
--format pretty|markdown|json
Terminal view by default; Markdown for docs, JSON for pipelines.
--no-color or NO_COLOR=1
Plain text output.
--model
Defaults to claude-sonnet-5-5, or set SOT_MODEL.
--provider auto|api|claude-cli
Auto uses the API when a key is set, otherwise Claude Code.

Honest by construction

A tool that judges AI claims has to hold itself to the same bar. These are the rules it is built on.

  • Inconclusive beats invented

    No usable evidence means no score for that dimension. Nothing usable at all means no verdict.

  • Adversarial on purpose

    Six skeptics argue against the claim and one steelman argues for it before anyone scores.

  • Validated at every hop

    Each agent must return JSON that passes a schema. One retry, then it fails loudly.

  • Fails soft, never fakes

    A failed skeptic drops its dimension and the weights renormalise. The run still finishes.

  • A rubric you can argue with

    Six dimensions and their weights are fixed and published, so every score can be checked by hand.

  • Every run is signed

    Scorecards record the model and the time they were made. Re-run a claim when new evidence lands.