C3 Measurement — Defect Catch-Rate, Not Completion

Learning objective

(Bloom's: Evaluate) Evaluate whether your agent-assisted review pipeline is actually working — by defining "working" in outcomes before collecting anything (defect catch-rate on payments-service, your own review minutes, false-flag burden), running a seeded-defect drill against ground truth you planted, and rendering a keep/change verdict from your own pre-registered decision rule — AIHC.2.C3 (Measurement).

Work at hand: your pipeline is built (L2–L6), calibrated (L7), recorded (L8), and proposed to the team (L9) — and if a teammate asked "is it working?" your honest answer today is "it ran." Lesson 1 measured you; nothing yet has measured the pipeline. Your L9 at-bat ended on exactly this dangling sentence. Today you close it the only credible way: numbers you designed before you liked them.

SSD cycle 1 — define "working" before you look: completion flatters everyone

• SAY: The lazy metrics are completion-shaped: PRs processed, reviews run, hooks fired. They measure that the pipeline happened, not that it helped — a first-pass reviewer that catches nothing still "processes" every PR (Playbook §9: completion and satisfaction are near-invalid proxies; measure transfer and outcomes). C3 at band 2 means your definition of "working" is written down, in outcome units, before data collection starts — because a definition written afterward will quietly bend toward whatever the data flattered [instructor synthesis — measurement design, not product behavior].
• SEE: Two views of the same imaginary fortnight, side by side (illustrative numbers, labeled as such): COMPLETION VIEW OUTCOME VIEW 14 PRs processed 100% seeded defects caught at first pass 3 / 5 first-pass ran 14/14 escaped to CI or later 2 (both item-4 class) hooks fired 41 times your median review time per PR 11 min → 6 min "pipeline adopted ✓" false flags you had to dismiss 4 (all item-6 class) The left view supports a slide; the right view supports a decision — it says what to fix (item-4 coverage), what to keep (the time win), and what's taxing you (item-6 noise). (Fania's Yankee Stadium record isn't great because the band got through the setlist — finishing the show is the completion view; what the crowd did during "Congo Bongo" is the outcome view.)
• DO: Write your pipeline's definition of working as three numbers with thresholds, in your own units and your own honest floor — e.g. catch-rate ≥ X/5 on seeded drills, median review ≤ Y min, false flags ≤ Z per week (values yours, not these). Add one line per threshold: why this number and not a flattering one. This document is pre-registration; date it.

SSD cycle 2 — baseline first: you can't measure a delta from a vibe

• SAY: "The pipeline made review faster" needs a before. You have one — your pre-pipeline merges (L1's #241/#238/#235 era) — but its numbers were never logged, so the baseline must be reconstructed and labeled as estimate: an honest measurement design distinguishes recalled data from logged data forever after (design-before-collection — AIHC C3; the discipline of marking estimate-vs-logged is [instructor synthesis]).
• SEE: The baseline table, half-filled the honest way: PR review minutes defects found by you defects found later basis #241 ~15 (recalled) 1 1 (the fee bug!) ESTIMATE — memory #238 ~10 (recalled) 0 0 ESTIMATE — memory #235 ~25 (recalled) 2 1 ESTIMATE — memory + issue log next 5 PRs (to be logged live) LOGGED — timer + record Note what row 1 quietly establishes: your manual process let the fee bug through. The baseline isn't a golden age to defend; it's the thing the pipeline has to beat.
• DO: Reconstruct your real three-PR baseline (minutes, defects-you-caught, defects-found-later — mine the issue tracker and your L6 failure log for the "later" column), mark every cell ESTIMATE or LOGGED, and start the live half: a one-line log entry per PR from today (start time, end time, verdicts). Keep it in docs/review-metrics.md next to the failure log — measurement that lives nowhere is measurement that stops.

SSD cycle 3 — seeded drills: catch-rate against ground truth you planted

• SAY: Waiting for real defects to measure catch-rate is slow and unfalsifiable — when the reviewer reports "no issues," you learn nothing about whether there were none or it missed them. The fix is planting known defects and counting recoveries: ground truth you control, which is what makes "the reviewer flagged 9 things" convertible into a rate (X8: the reviewer agreeing with itself, or with a second agent, still isn't evidence — the seed list is) [instructor synthesis — drill design; the isolation mechanism below is frozen product fact]. Isolation matters: drills must never touch real branches — §3: claude --worktree feature-auth "creates an isolated git worktree/branch so two terminals don't collide; repo needs ≥1 commit first" — so the drill lives in a worktree and dies with it.
• SEE: Your five-seed list, one per risk class you actually govern, each mapped to the checklist item that claims to cover it: S1 float arithmetic in a new fee helper → item 1 (Decimal rule) S2 customer email added to a log line → item 4 (PII) S3 hard-coded token in a test fixture → item 6 (secrets) S4 FEE_RATE changed with value preserved-then-not → L5 leg (b) (must-not-change) S5 endpoint response field renamed, docs untouched → item 2 (api-contract diff) Five seeds, five different detection mechanisms on trial — a drill that seeds five variants of the same bug measures one mechanism five times and calls it a pipeline test.
• DO: Build your real seed list (these five or your own, one per mechanism), plant them in a --worktree drill branch, run the full pipeline on the drill PR exactly as if live (reviewer first-pass, your verification legs, hooks armed), and score each seed CAUGHT-at-first-pass / CAUGHT-by-your-legs / ESCAPED — the three-way split matters, because "the pipeline caught it" hides which layer did.

SSD cycle 4 — the decision rule: pre-commit to what the numbers will make you do

• SAY: Numbers without a pre-committed consequence become decoration. The C3 closing move: write the if-then before the fortnight's data is in — and make the first "then" a scaffolding move, not a verdict on the model: when a layer underperforms, X11 says the first hypothesis is a gap in what you provided (contract wording, reviewer scope, docs/pii-fields.md coverage), and only after a scaffold fix fails does the conclusion escalate. Small-n honesty belongs in the rule too: one fortnight adjusts; it doesn't conclude [instructor synthesis].
• SEE: A decision rule in the required shape (skeleton — yours must carry your cycle-1 thresholds): IF catch-rate < threshold AND misses cluster in one item class → treat as scaffolding gap: tighten that item's contract/description; re-drill in 1 wk IF review time > threshold → find the minutes: false flags? verification legs? → fix the top consumer, re-measure IF false flags > threshold → narrow the flagging rule's scope (lane, matcher, or reviewer wording); re-measure IF all three inside thresholds two fortnights running → propose L9 table go team-default; measurement drops to monthly drills
• DO: Write your real decision rule against your cycle-1 thresholds, date it, and append it to the pre-registration doc — then render today's verdict from the drill you ran in cycle 3: which branch of your own rule fired, and what did it obligate you to do before the next drill? Do the obligated thing (or schedule it, with a date, if it's bigger than today).

Independent at-bat

Unscaffolded, and it runs for two weeks: operate the measurement system you just built — live log every real PR, one drill per week in a fresh worktree with a rotated seed list (same mechanisms, new instances; a static seed list trains the drill, not the pipeline), and at the fortnight's end write the four-line outcome report in docs/review-metrics.md: three numbers vs. thresholds, the decision-rule branch that fired, the action taken, and the one number you now distrust and why. No template beyond what cycles 1–4 already made you write for yourself.

Exit ticket

(Q1–Q2 graded against your pre-registration doc and the cited sections; Q3–Q5 are [instructor synthesis]-scoped — graded against your own written thresholds, seed lists, and decision rule, the only ground truth measurement has here.)

1. (Remember) Name your three outcome measures, their thresholds, and where the pre-registration doc lives.
2. (Understand) Per §3, why does the drill run in a --worktree — and separately, why must the seed list exist before the reviewer runs rather than being reconstructed after?
3. (Apply) Fortnight one comes in: catch-rate 4/5 (miss = S2, the PII seed), median review 7 min, false flags 6 — all against your real thresholds. Apply your own decision rule: which branch fires first, and what exactly do you change?
4. (Analyze) Your reviewer caught S3 (secrets) every drill, and a second agent you added for confirmation also caught S3 every drill. A teammate calls that "double-verified." Using X8 and your three-way scoring split, analyze what the agreement does and doesn't add — and name the one measurement that would justify keeping the second agent.
5. (Evaluate — objective level) Render the verdict your pre-registration demands: is the pipeline working, per your own thresholds, on today's evidence? State the verdict, the weakest number under it, and the single next measurement that would most change your confidence — in either direction.

ledger_write

ledger_write:
  learner_id: L1-DEV-MAYA
  lesson_id: L1-dev-maya-10-c3-measurement
  anchors: [AIHC.2.C3]
  bloom_reached: Evaluate
  exit_ticket_score: ""
  auto_score: ""
  self_score: ""
  calibration_gap: " — computed once Maya's self_score exists"
  starting_mastery:
    AIHC.2.C3: "not previously evidenced — first measurement this lesson"
  journal_prompt: >
    Before cycle 2 you believed something about whether the pipeline was saving you time.
    Write down what you believed, then the reconstructed baseline number, then the first
    logged number — and one sentence on which of the three you'd have defended in standup
    last week without noticing it was a vibe.
  structure_used: PBL
  referents_used: [salsa-fania-yankee-stadium]
  next_lesson_seed: >
    Lesson 11 scales out: the CI migration itself becomes a multi-agent orchestration —
    D1, her weakest profiled anchor (0.30, band 1) — with the measurement system from this
    lesson watching whether orchestration helps or just multiplies.

RUBRIC SELF-AUDIT

Escalation: all load-bearing indicators PASS → auto-ships (Playbook §5, AIHC X2/B3).