B2 Failure Literacy — A Failure Log For The Agent-Assisted Pipeline

Learning objective

(Bloom's: Analyze) Analyze the characteristic ways your agent-assisted review pipeline can fail — context loss, permission surprises, and ungrounded references (hallucinated paths/claims) — by tracing each to the frozen mechanism that permits it, and institute a running failure log for payments-service in which every entry crystallizes into a rule, a hook, or a tightened contract — AIHC.2.B2 (Failure literacy).

Work at hand: lesson 5 ended with two live findings nobody has filed anywhere: the FEE_RATE must-not-change wording ambiguity, and the missing rounding-mode note. Today they become entries 1 and 2 of payments-service/docs/failure-log.md — and the log becomes the PFI-style habit your verifier role runs on: every failure documented as a proposed failure-mode incident (trigger → observed behavior → gap → rule), before the fix, so the pipeline learns instead of just you.

SSD cycle 1 — context loss: your rules are context, not law, and sometimes not even loaded

• SAY: The standing specification you wrote in lesson 2 (the CLAUDE.md Decimal rule) can fail in two distinct ways, and both are documented mechanisms, not bugs: it can be loaded and not followed, or not loaded at all. Failure literacy means knowing which one you're looking at before you write the fix.
• SEE: The two mechanisms, from the doc-set, applied to your rule: LOADED BUT SOFT §4 Memory: CLAUDE.md is "context, not enforced config"; it has "no compliance guarantee," and "contradicting rules across files get resolved arbitrarily." Diagnosis tool: /context shows what actually loaded. NOT LOADED AT ALL §8 Subagents: "Explore and Plan are the two exceptions that skip loading CLAUDE.md" — so any research you delegate to those two built-ins runs WITHOUT your Decimal rule in view. Every other built-in and custom subagent loads it. Your payments-first-pass-reviewer loads the rule; a quick Explore errand does not. Same repo, same rule, different truth — and neither is a model deficiency; both are documented scope you hadn't yet mapped (AIHC X11: read the unmet expectation as a scaffolding gap to fill, not a defect to blame).
• DO: Create payments-service/docs/failure-log.md and file entry 1 — lesson 5's missing rounding-mode note — in four fields: trigger (what was asked), observed (what came back), gap (which mechanism above explains it — was the rule loaded? run /context in the session and record the answer), rule (leave blank; cycle 3 fills it). Then check your own habits: list which of your delegations from lessons 3–5 ran through built-ins that skip CLAUDE.md, and note whether any conclusion you kept came from a session your rules never reached.

SSD cycle 2 — permission surprises and ungrounded references: the other two entries

• SAY: Two more characteristic failure shapes, one grounded in frozen permission mechanics, one in reference discipline. Permission surprises: the permission behavior you set on a subagent is not absolute — §8: a subagent's permissionMode can be overridden, because "a parent session in bypassPermissions/acceptEdits takes precedence and can't be overridden"; and §9: rules merge across settings scopes with deny always winning, and project-level allow rules need a one-time workspace-trust approval — so a teammate cloning payments-service gets prompts you stopped seeing months ago. Ungrounded references: an agent report can name a path or test that does not exist in your repo — the doc-set's grounding tools are the countermeasure: @file inlines the actual file (§3), and lesson 5's rule already covers the rest: a named check gets run, not believed. The failure-mode label "hallucinated paths" itself is [instructor synthesis] — the doc-set documents the grounding mechanisms, not a failure taxonomy; the taxonomy move comes from AIHC B2/P7.
• SEE: Both shapes as they would land in your pipeline, one line each: PERMISSION SURPRISE Your reviewer runs inside your acceptEdits parent session → its own prompt behavior yields to the parent's (§8). The read-only guarantee you're relying on is its tools: allowlist (Read, Grep, Glob — no edit tools granted), not its permissionMode. UNGROUNDED REFERENCE First-pass report cites tests/payments/test_fees.py as evidence. Your repo's file is test_fee_calculation.py. The citation is load-bearing and false; @-check would have caught it in one line.
• DO: File entries 2 and 3 in your log: entry 2 = lesson 5's FEE_RATE wording ambiguity (a contract gap, filed under specification failures — gap field names your own contract, not the agent); entry 3 = a permission-or-grounding failure from your own sessions this week (pick a real one; if genuinely none occurred yet, file the §8 parent-precedence surprise as a proposed failure mode you now know to watch for — that's the PFI habit: proposed incidents count, labeled as proposed).

SSD cycle 3 — crystallize: every entry ends in a rule, and rules have three strengths

• SAY: A failure log that only describes is a diary. The B2 habit closes each entry by choosing the weakest mechanism that actually prevents recurrence, and the doc-set gives you exactly three strengths [instructor synthesis, from §10]: CLAUDE.md (persistent but soft — §4), a tightened contract check (runs when you run it — lesson 5), or a hook (§6: hooks exist to ensure "certain actions always happen rather than relying on the LLM to choose to run them" — the only deterministic one, AIHC X4).
• SEE: The doc-set's own hook example (§6, verbatim) next to the crystallization it suggests for your entry 1: json { "hooks": { "PostToolUse": [ { "matcher": "Edit|Write", "hooks": [ { "type": "command", "command": "jq -r '.tool_input.file_path' | xargs npx prettier --write" } ] } ] } } Entry 1 (rounding-mode note keeps going missing) is a candidate for the same PostToolUse shape: after any edit under src/payments/, run the contract's test file automatically — the note requirement stops depending on anyone remembering it. Entry 2 (contract wording) is NOT a hook problem: it crystallizes as tightened contract text. Matching the fix's strength to the failure's mechanism is the analysis this lesson is named for. (A band that fell apart in the same break twice doesn't rehearse the whole chart again — the bandleader marks that bar and drills the break itself.)
• DO: Fill the rule field for all three entries: name the mechanism chosen (CLAUDE.md line / contract check / hook), the one-line content of the rule, and why this strength and not a stronger or weaker one. Draft the actual hook JSON for whichever entry you assigned one (adapt the §6 example; put it in .claude/settings.json per §6's configuration locations so it's shared with the team).

Independent at-bat

Fully unscaffolded, and it closes your arc: run one real PR through the full pipeline you have now built — contract (L2), scoped delegation (L3), elicitation checkpoint (L4), four-leg verification record (L5) — and when anything fails anywhere in that run, file it in the log and crystallize it before you fix it. If the run comes back clean, file that too, with the evidence line that makes "clean" a checked claim rather than a vibe. This at-bat is your profile's stated learning goal ("run one real PR through an agent-assisted review I actually trust") — the log entry is what "actually trust" now means: trusted because instrumented.

Exit ticket

(Graded against your failure log, your hook draft, and the cited doc-set sections — never against an agent's account of why it failed.)

1. (Remember) Name your log's three entries and the four fields each one carries.
2. (Understand) Per §8, your reviewer's read-only behavior rests on its tools: allowlist rather than its permissionMode. Why does the parent session's mode make that distinction matter?
3. (Apply) Next week the reviewer cites a checklist item number that doesn't exist in review-checklist.md. Write the complete four-field log entry, including the crystallized rule and its chosen strength.
4. (Analyze) Entry 1 got a hook; entry 2 got contract text. Trace each choice back to the mechanism that permitted the failure, and explain why swapping the two fixes would leave both failures able to recur.
5. (Evaluate) Judge your pipeline after the at-bat: which single log entry, if its rule were deleted tomorrow, would make you stop trusting the pipeline first — and does that answer change where you spend your next hardening hour?

ledger_write

ledger_write:
  learner_id: L1-DEV-MAYA
  lesson_id: L1-dev-maya-06-b2-failure-literacy
  anchors: [AIHC.2.B2]
  bloom_reached: Analyze
  exit_ticket_score: ""
  auto_score: ""
  self_score: ""
  calibration_gap: " — computed once Maya's self_score exists"
  starting_mastery:
    AIHC.2.B2: "not previously evidenced — first measurement this lesson"
  journal_prompt: >
    Before this arc, a pipeline failure meant "the agent messed up." Reread your three log
    entries: how many gap fields actually name the agent, versus your contract, your rule
    placement, or a documented mechanism you hadn't mapped? Write what that ratio does to
    how you'll talk about the next failure in standup.
  structure_used: PBL
  referents_used: [salsa-fania-bandleader]
  next_lesson_seed: >
    Arc complete — learning goal at-bat taken. Next arc candidate per ledger: AIHC.1.D1
    (orchestration, still band 1 / 0.30) once the single-reviewer pipeline is stable; or a
    B2 follow-up at Evaluate if the at-bat's log entries show mechanism-tracing still shaky.

RUBRIC SELF-AUDIT

Escalation: all load-bearing indicators PASS → auto-ships (Playbook §5, AIHC X2/B3).