The library

Rendered from docs/specs/2026-09-26_post-draft-heads.md; the markdown file is the record.

Run spec: post-draft heads lane, frozen WavLM Large probe (Sat Sep 26, 2026, 18:15)

Rung: post-draft heads, from docs/handoffs/2026-09-26_post-draft-heads-handoff.md (revised ~17:25: the XLS-R layers 5–9 probe is dropped, since Fable ran it on the on-disk embeddings and it failed on In-the-Wild). P1 frozen by Nathan ~17:35 and P2 ~18:05, compressed process: no code plan, so no Codex plan review; the Claude critique and Codex audit run on this spec and the report after the export lands. The 18:50 hard stop was lifted by Nathan (~18:10); the export is due ~19:15, before the gate packet at 19:30. Nothing here touches fusion, the runner, the TSV or any shipped file.

This spec was written at ~18:15, before any WavLM probe number was seen; the checks and the fallback below are pre-declared. Text added after the run is marked (post-run): D5's diagnostics clause, the Results section and the Post-run notes section. D4's diagnostics existed before the probe ran as a scratch script (heads_diag.py, 18:14; its self-check against the reference counts ran at ~18:15); it was committed as scripts/heads_diagnostics.py after the run.

Problem

NSA's draft review returned minDCF 0.0733 on the shipped rule (A3 w0.2 + E); one team is ahead at 0.0584. The shipped rule misses 158 of 1,000 In-the-Wild fakes at its In-the-Wild brief argmin (15 FA). Of those, M1b alone flags 1 at its inner threshold, handcrafted 2, M5 33, Spectra 107: heads on the XLS-R backbone share M1b's blind spot. A different frozen backbone is the one head experiment left. The expected test effect is near zero; a clean negative result is graded README content.

What runs

Design decisions

• D1 (recipe = M1b v3). Segment mode, band match, silence trim, 8 s cap, per-input normalization (hearsay.embed; WavLM's own do_normalize: true is the same operation, so nothing differs), time-mean per layer, all 25 hidden states stored in fp16. Crops: test-length, seed 0 for the NSA train sample, seed 100 for the ASV19 add-on (what asv19_addon_v3/extract_meta.json records; the handoff's command said 0), none for the test set, seed 200 for In-the-Wild. Probe: layer by inner-CV brief minDCF only; StandardScaler + class-balanced logistic (C = 1) + Platt, all fold-local; ASV19 rows are inner-only; holdout read once; In-the-Wild never fit on. No code in the recipe changes.
• D2 (parallel extraction and decode repair). To fit the time box, the four sets were extracted by three concurrent processes after the serial chain was cut (the training sample resumed from its five completed shards; resumes are bit-identical because crop lengths and per-row seeds are regenerated deterministically). Under that load FFmpeg failed transiently on files that decode fine on retry (30 rows by 18:10; 0 in every v3 set; 0 in the five serial shards). scripts/repair_decode_errors.py re-embeds only flagged rows through the extractor's own segment path (stored crop length, seed = meta seed + row), after an identity check that re-embeds unflagged rows of the same set and refuses to write if they do not reproduce (relative error ≥ 2e-3). Run on each set only after its extraction has finished.
• D3 (fallback, pre-declared). If the ASV19 add-on is not extracted when the training sample is, train NSA-only, call it an M1-style probe, and compare it against M1 v3 (holdout 0.159, In-the-Wild 0.380), not M1b. This restates the handoff's fallback with a different trigger. The handoff's trigger ("train sample not done by 18:10") did fire: the train sample finished at 19:04. It was waived because Nathan lifted the 18:50 hard stop (~18:10) and then asked for the full NSA + ASV19 recipe unless the add-on actually failed (relayed by the oversight chat, ~18:55).
• D4 (diagnostics, pre-declared). (a) Spearman of wavlm_l vs m1b_v3 logits on holdout, test and In-the-Wild; reference Spectra vs M1b on test (0.62 by this lane's computation; the handoff quotes 0.57). (b) Miss tail: reproduce the shipped rule on In-the-Wild from the exports (must give exactly 15 FA / 158 miss at the ITW brief argmin, else stop); each column's inner threshold = brief-cost argmin over its own inner-OOF logits; count of the 158 each column alone flags. Reference, reproduced by this lane's script: M1b 1, handcrafted 2, M5 33, Spectra 107. No fused numbers: the gate chat computes those.
• D5 (ownership). Tracked changes: this spec, the report, scripts/repair_decode_errors.py, tests/test_repair_decode_errors.py and (post-run) scripts/heads_diagnostics.py (D4's diagnostics, promoted from a scratch script after the run so every reported number cites a tracked file; it reads exports only and hard-asserts the 15 / 158 reproduction). No change to embed.py, probe.py, the extraction, training or export scripts, the fold files or any fusion file. Staged by path; not pushed.

Acceptance checks (fail → stop and report)

tests/test_repair_decode_errors.py covers D2's tool (6 tests):

• flagged rows only, with the stored crop and the row seed
• NaN crop for uncropped sets
• identity mismatch refuses and writes nothing (fails if the check is removed)
• a still-failing row stays flagged
• a no-op on clean sets
• windows-mode refusal

Readouts reported (no pass/fail: the gate chat judges)

CV-by-layer table; holdout minDCF, EER, act DCF, per-generator and per-source; In-the-Wild minDCF (brief) and averse cost, P_FA and P_miss at the inner threshold; Spearman (D4a); miss-tail count (D4b); timing. For the column alone, the interesting outcomes are In-the-Wild below M1b's 0.342 and a test-set Spearman with M1b well below 1.

Results (19:10) (post-run)

Full readouts, CV table, diagnostics and the decode-repair incident are in docs/reports/2026-09-26_post-draft-heads.md. Summary:

Run.

• Layer 9 by inner CV (0.2936).
• Holdout 0.0717 / EER 1.45%, level with M1b v3.
• In-the-Wild 0.752 brief / 0.383 averse, against M1b's 0.342 / 0.296. P_FA at the inner threshold is 7.05%, against 0.8%.

Diagnostics.

• Spearman vs M1b: test 0.795, holdout 0.863, In-the-Wild 0.774.
• Miss tail: WavLM flags 57 of the shipped rule's 158 In-the-Wild misses. References: M1b 1, handcrafted 2, M5 33, Spectra 107.

Checks.

• A1–A9 pass. 99 transiently failed rows were repaired, identity relative error 0.0, and none are still failing.
• A10 at e611d1b: pytest 745 passed (run on the working tree at ~19:08, before other lanes' later commits). The first version of this section said repo-wide ruff failed on src/hearsay/analyzer.py from 19925a3. That was stale: those findings were fixed in 6a1ba6b and 1b66b56 before e611d1b, and ruff check . is clean there. Counts after the audit fixes are in Post-run notes.

Deviation from D2's plan. The three-process layout was abandoned at ~18:15 because of swapping, and the train-sample process was restarted clean at 18:34. Neither changes any output: resumes are deterministic, and every repaired set passed the identity check.

Gate verdict (gate chat, commit 57b9c90): the pre-declared entry failed both the gate and the room. WavLM does not ship; the result is recorded as a negative.

Post-run notes (post-run; Claude pre-audit critique round 1, Fail → fixes)

The critique graded e611d1b Fail on Test coverage and Documentation. Fixed in the follow-up commit:

• A5 evidence was persisted for one set only, and the identity bar was lax.
  • The first bar was a whole-matrix max relative error, so a WavLM row with values near 500 allowed ~1.0 absolute error per element. It could also pass with zero rows checked.
  • scripts/repair_decode_errors.py now checks element-wise (|diff| ≤ 5e-3 + 5e-3·|stored|). It requires exactly --verify ≥ 1 unflagged rows, drawn at random from the flagged shards first and then from the others.
  • It refuses a --manifest that differs from the set's manifest.csv, and a set whose extraction has not finished. It writes shards atomically.
• The --verify-only mode replaces the untracked finish-chain validation (A1–A5). Run on all four sets after training, it re-embedded 426 rows (10 per shard plus all 99 repaired rows), all matching exactly (max abs diff 0.0). Stored crop lengths equal the v3 twins'. Evidence: outputs/logs/wavlm_verify_*.json.
• D2 changed accordingly. The tool described in D2 is the tightened version; the repairs themselves were made by the first version, and the verify-only run above re-checks their output under the new bar.
• Tests. tests/test_repair_decode_errors.py now has 17 tests. The new ones cover the empty pool, n_verify = 0, fallback to other shards, a mismatched manifest, an unfinished set, an element-wise tolerance pin at WavLM scale, no leftover temp file, embed_row through the real prepare_segment / embed_segment with a stub model, and verify-only pass and fail cases.
• Documentation.
  • A10 corrected.
  • A submissions/log.csv row added (no TSV; not shipped).
  • The report: repaired-row IDs, A5 evidence, extraction start 17:19, per-shard solo rates, the inner-OOF averse gap, and the Reading section reconciled with the gate verdict.
  • The CLAUDE.md gotcha line: 99 of 32,319 rows (0.31%), flagged rather than silent. The AI-use disclosure's WavLM entry had already been updated by other lanes (1dccb8c, cd9eb11); a heads-lane bullet was added.
• A10 after the fixes: uv run pytest -q 770 passed, 1 skipped (working tree including other lanes' commits up to 1461c1f); uv run ruff check . clean.
• Housekeeping. The submissions/log.csv row was appended in this lane but landed in the gate lane's commit cbee709, which staged the file after the append. cf11b85's message says it added the row.

Post-run notes, round 2 (post-run; Claude critique round 2: Acceptable, low items fixed)

• N1. The atomic write's temp file is now .shard_XXXXX.npz.tmp, outside the shard_*.npz glob, so a crash cannot leave a second copy of the rows for a loader to pick up.
• N2. verify_set now reports only the explicit rows it actually re-embedded, and refuses an empty check.
  • Five tests were added: explicit rows really re-embedded, twin meta mismatch, twin path mismatch, a manifest-only flag, and a non-finite embedding. Plus a failed rename that leaves the original shard untouched.
  • tests/test_repair_decode_errors.py now has 22 tests. The critique's 17 single-guard mutants of the tool are all caught (6 survived before).
• N3. The report's test count is updated.
• N5. The report states the source of the 25 recovered row IDs.
• N4. The CLAUDE.md lane bullet names a Codex audit; its result is recorded below once it has run.

Post-run notes, round 3 (post-run; Codex audit round 1: Fail → fixes)

• Calibration wording (Plan adherence and Documentation, Fail). D1's "StandardScaler + class-balanced logistic (C = 1) + Platt, all fold-local" copied the handoff's summary of the M1b recipe, and it is wrong for Platt.
  • scripts/train_probe.py:78 fits one Platt map on the pooled inner out-of-fold scores, and scripts/export_probe_scores.py:56 applies it back to the inner rows. The scaler and classifier are fold-local.
  • The run mirrored M1b exactly, which was D1's intent. The claim is corrected and the recipe is unchanged: changing it would break the like-for-like comparison with M1b and the gate's column.
  • Platt is a positive affine map. Ranks, Spearman figures, rank-fusion inputs, argmin thresholds and every minDCF or EER in the report are unaffected; only calibrated inner logit values are in-sample.
  • Corrected in the report's recipe section and in docs/architecture.md's validation diagram, whose inner_oof edge made the same claim for every M1-family column.
• Final suite (re-run after this round's changes, working tree at HEAD f085d89 plus this round): uv run pytest -q 775 passed, 1 skipped; uv run ruff check . clean.

Post-run notes, round 4 (post-run; Codex audit round 2: Documentation Fail → fix)

• docs/architecture.md's fold-discipline invariant still said every learned piece, "including … Platt maps", is fit fold-locally.
  • It now states the exception: the M1-family and fusion Platt maps are fit once on pooled inner out-of-fold scores, so ranks and metrics are unchanged and calibrated inner values are in-sample. No outer-holdout, stress or test label enters any fit.
  • The diagram edge at the inner_oof column says the same.
• Docs-only change. The suite result above stands; tests/test_docs_consistency.py was re-run on the edit.

Review scorecards

The round-5 dimension grades were:

Codex could not re-run the full suite in its read-only sandbox. It passed the docs tests (52/52) and ruff, and it reproduced the diagnostics, the export counts and the shipped TSV hash itself.