The library

Rendered from docs/reports/2026-09-26_fusion-sweep-predeclared.md; the markdown file is the record.

Fusion sweep: pre-declared candidates and selection rule (written before any result, Sat Sep 26 ~08:15)

Per the fusion consult response (docs/consults/2026-09-26_fusion-strategy_RESPONSE.md).

Candidates

Written before running:

• A. α-sweep (rank domain). score = (1 − α)·rank(M1b v3) + α·rank(handcrafted v5), for α ∈ {0, 0.1, 0.2, 0.3, 0.4, 0.5}. Ranks are each file's ECDF position against that detector's own inner OOF distribution.
• B. min rule and max rule over the two ranks.
• C. Cascade. M1b rank everywhere, except in the ambiguous middle band (M1b rank between the inner 40th and 90th percentiles), where it is replaced by the mean of the M1b and handcrafted ranks.
• D. Non-negative stacker shrunk toward equal weights, over z(M1b) and z(handcrafted). Fit on inner OOF with LJ bona fide excluded; weights clipped at ≥ 0, then 50/50 shrinkage toward equal.
• E. M3 as false-alarm suppression only, applied on top of the winner of A–D. Where M3's margin is below −3 ("strongly bona fide") and the base rank is above 0.5, the base rank is multiplied by 0.5. M3 is never used to raise a score, and nothing is fitted on M3.

Readouts, for every candidate

• Inner OOF minDCF, brief cost (π_synth = 0.3, C_FA = 4).
• Holdout minDCF under both costs, plus the raw (#FA, #miss) at each candidate's holdout argmin threshold.
• Per real source: LJ and LibriSpeech.
• In-the-Wild minDCF under both costs:
  • brief cost: 9.33·P_FA + P_miss
  • miss-averse cost (the sponsor code's semantics): P_FA + 4·P_miss at π 0.5
• In-the-Wild P_FA and P_miss at the inner-OOF threshold.

Selection rule (fixed now)

1. Admissible: inner OOF within 0.03 of the best candidate's inner OOF, and In-the-Wild miss-averse minDCF ≤ 0.45.
2. Among admissible candidates, pick the lowest In-the-Wild brief-cost minDCF. Ties within 0.01 go to the smaller α, meaning more M1b.
3. Apply E only if it lowers In-the-Wild brief-cost minDCF by ≥ 0.01 and does not raise holdout or inner OOF by more than 0.01.
4. No iteration after the results. If nothing is admissible, ship M1b v3 alone.

Results (appended after the run, ~08:13; the rule above was not changed)

Admissible: α 0.2–0.5, C and D. The lowest ITW brief-cost score is α 0.2 and α 0.3, tied at 0.322 with D at 0.324 inside the 0.01 band. The tie goes to the smaller α, so the winner is α 0.2, inside the consult's predicted 0.2–0.3.

E (M3 as false-alarm suppression only) lowers ITW brief from 0.322 to 0.260 and does not raise the holdout or inner scores, so it is applied. FINAL = E on α 0.2.

Caveat. E's inner, holdout and ITW gains all involve M3, whose training data is undisclosed. E can only move a file toward "real", so if M3 is wrong about a file, the cost is a miss (weight 1), never a false alarm.

Caveats added 13:10 (channel-robustness lane, relayed by the oversight chat). (1) The M3 leakage probes returned KEEP for the suppression step: M3's margins are the most stable under perturbation (mean |ΔAUC| 0.0008), though its rank order is the least stable of the three deep models (clean-vs-perturbed Spearman under noise: M3 0.875, M1b 0.913, M5 0.939); it fires on no holdout fake under any perturbation (it damps 1.2% of MLAAD fakes, 7 of 572) and never raised fused minDCF, and it had the lowest MLAAD miss rate; the suppression step itself is safe. (2) The rule is not noise-robust: 20 dB additive white noise collapses handcrafted v5 (AUC 0.998 → 0.64) and takes the shipped rule's holdout from 0.012 to 0.269 while M3 holds; the test set shows no additive noise, no rule change was proposed, and symmetric noise augmentation is future work.

Files:

• Primary: submissions/20260926-0813_M4_sweep_E_on_A_alpha0.2_our_direction.tsv
• Flipped (use only if NSA's feedback decodes as their code's polarity): ..._FLIPPED_only_if_NSA_scores_inverted.tsv
• Constants for the runner: models/fusion_v1/constants.json
• Speech gate: 0 of 1,671 test files gated. Determinate scores sit in [0.001, 1].

Addendum: M5 candidates, pre-declared before any run (Sat ~08:35, requested by Nathan via the oversight chat)

M5 = outputs/detector_scores/m5_xlsr_ft.csv. It is the frozen-backbone, head-only arm; its inner rows are out-of-fold. Its In-the-Wild scores are the 2,000 real clips already scored plus the 1,000 spoof clips scored now with scripts/m5_score.py, using the same seed-200 crops.

F: M5 as a second false-alarm suppressor, on top of the shipped E-on-A-α0.2 rule.

• Order: after the M3 step. If m5_logit < t_F and base > 0.5, then base *= 0.5. M5 never promotes a file toward synthetic.
• Threshold: t_F is fixed from inner OOF only, before the holdout or In-the-Wild is looked at. It is the M5 logit value below which 1% of inner spoof rows fall, so the suppression can fire wrongly on at most ~1% of known fakes.
• Why that reading: the request said "~1% inner-real coverage", which is ambiguous. A threshold that fires on only 1% of real rows would almost never act, so this is the conservative interpretation that keeps the step meaningful. The fraction of inner real rows it covers is reported alongside.

A3: three-way rank blend with weights (0.8 − w, 0.2, w) for (M1b, handcrafted, M5).

• w ∈ {0.1, 0.2}, each run without and with the M3 suppression step (same E rule as before), so four variants.

Readouts: identical to the 08:13 sweep.

Decision rule (fixed now). The current rule (E on A α 0.2) is replaced by a candidate only if all of these hold:

1. In-the-Wild brief-cost minDCF improves by ≥ 0.01.
2. In-the-Wild miss-averse minDCF gets worse by no more than 0.01.
3. Holdout brief-cost gets worse by no more than 0.05 (inside the holdout's noise band).
4. Inner OOF gets worse by no more than 0.03.

If several candidates qualify, the one with the best In-the-Wild brief-cost wins. If none qualifies, the rule stays frozen and the negative result is recorded here. Any change needs Nathan's ratification before it ships.

Expectation, stated in advance: F may buy a little on In-the-Wild false alarms. A3 will probably lose on the holdout's short clips and LibriSpeech real.

Addendum results (appended after the runs, ~09:15; the rule was not changed)

Deviation from the addendum, disclosed. The addendum said M5's In-the-Wild spoofs would be scored "using the same seed-200 crops". scripts/m5_score.py has no crop option, so the first run scored all 3,000 M5 In-the-Wild clips whole, capped at 8 s, while the other detectors saw seed-200 test-length crops. The run was then repeated crop-fair: the identical prepare_segment(x, crop_s, 200 + row) crops were written to WAV and scored with m5_score.py. Both runs are reported. The crop-fair run is the decision input.

t_F = −3.90 (1% of inner spoof rows fall below it; it covers 41.8% of inner real rows).

Decision under the pre-declared rule: A3 w 0.1 + E and A3 w 0.2 + E both qualify; the winner is A3 w 0.2 + E. The weights are 0.6 M1b v3 + 0.2 handcrafted v5 + 0.2 M5 in rank space, followed by the M3 false-alarm suppression step. F does not qualify: its ITW gain of 0.003 is under the 0.01 bar.

The advance expectation was wrong. A3 was expected to lose on the holdout's short clips and LibriSpeech real, but with the M3 step it improved both (LibriSpeech 0.0186 → 0.0065).

Effect on the test set (current rule vs candidate): Spearman 0.974, and only 3 of 1,671 files cross 0.5.

Status: SHIPPED (Nathan, 12:20: GO A3; the draft went to NSA at 12:03 as CrossExam_predictions.tsv = the 09:14 our-direction file below; runner default switched 12:35). Until 12:20: candidate, pending ratification.

• TSVs, both polarities, are logged: submissions/20260926-0914_M4_sweep_A3_w0.2_E_CANDIDATE_*.tsv.
• Runner constants: models/fusion_v2_candidate/constants.json (moved at 10:45 to outputs/fusion/fusion_v2_candidate/constants.json, the record; the runner's file is models/fusion_v2/constants.json, see the 10:45 note below).
• Shipping it would require the pipeline and Docker to add the M5 scorer: a 657 MB checkpoint from the HF Hub, about 0.2 s per clip on CPU. (10:45: the pipeline half is done, see below; the image half waits for the switch.)

Nathan's decision (09:25): positive result, not shipped pending draft review.

• E on A α 0.2 stays frozen, and it is the file sent for the draft review.
• A3 w 0.2 + E is revisited only if NSA's draft-review number decodes to our direction and there is time left before 22:00.
• The candidate artifacts stay where they are. No pipeline changes. (Superseded at ~10:20 by the note below: Nathan approved building the runner side now, gated, so the switch is one line if the number comes back in our direction.)
• Reaction branches are in docs/reports/2026-09-26_draft-review-branches.md.

Runner side built, not switched (M4 fusion chat, 10:45; run spec docs/specs/2026-09-26_m4-fusion-v2-m5-scorer.md). Nathan (~10:20, "build now, switch later"): make the revisit a one-line switch without changing anything that ships. Done:

• scripts/fuse_sweep_m5.py --write (the flag now exists; without it nothing under models/ is written) wrote models/fusion_v2/constants.json: weights 0.6 / 0.2 / 0.2, the three inner-OOF rank references, the same M3 step, the Platt map, a how line, and the M5 checkpoint's hashes (m5_xlsr_ft_20260926-0741/model). Against the 09:14 candidate file: weights and references identical, Platt a and b identical to the last digit (delta 0.0). The candidate constants moved to outputs/fusion/fusion_v2_candidate/ so docker/build.sh stops staging them.
• The runner and the API execute it with --fusion models/fusion_v2/constants.json (M5 loads and runs only under a file that weights it; a checkpoint whose hashes differ from the file's is refused before anything is scored). Exports through the new file reproduce both 09:14 candidate TSVs on all 1,671 rows to 7e-16; live from audio on 50 template files: Spearman 1.0, max abs diff 2.6e-4 (the known CPU-vs-A100 M5 gap), 0 scorer errors. Full-set numbers in docs/reports/2026-09-26_runner-docker.md (v3).
• Unchanged at 10:45: the default rule, every TSV under submissions/, the draft-review payload, the Docker image. Switched at 12:35 (Nathan: GO A3 at 12:20 via the oversight chat, "switch" typed in the M4 chat at 12:35; NSA had said at ~12:00 that the image is not required): the runner's default is models/fusion_v2/constants.json; fusion_v1 stays loadable via --fusion as the fallback; the draft went out at 12:03 as the 09:14 A3 file; the full-set live run's parity to it is Spearman 1.0 and max abs diff 9.3e-4 on all 1,671 rows, both polarities (evidence under outputs/runner/v2_full/; the per-row M5-logit deviation on 5 rows was accepted by Nathan with the switch). The image stays on e_on_a as evidence; no rebuild.

Addendum 2: a refit M1b column, pre-declared before it exists (M4 fusion chat, Sat 12:50)

Why now. The channel-robustness lane measured λ̂ = 0.947 (CI 0.93–0.98): the test set reads as codec-processed audio, like In-the-Wild, not like the clean holdout. If that lane finds the codec and Nathan approves a symmetric refit of M1b (both classes round-tripped, per consult item 8), a new column may arrive around 16:00. This rule is written before that column exists and is not changed after.

What the candidate is. Exactly one: R1 = the shipped rule with the refit column in M1b's place. Rank space, weights 0.6 (refit M1b) / 0.2 (handcrafted v5) / 0.2 (M5), the same M3 false-alarm suppression (margin < −3 and base > 0.5 → ×0.5), the same Platt map fit on inner OOF. No weight re-sweep, no second candidate, no iteration: the refit changes one detector, so the question is only whether that detector's column is better in the same seat.

What the column must be, or the sweep is not run.

• outputs/detector_scores/<name>.csv in the standard export format (path, fold, split, score, logit; 21,671 rows; inner OOF from the same fold file, holdout scored once by a model that never saw it) plus its In-the-Wild rows on the seed-200 crops (as split stress in the same file or _itw_<name>.csv), because the M5 whole-clip deviation this morning showed why.
• A report that states the augmentation was applied to both classes and shows the clean-only holdout score beside the augmented one (the model rule); a refit whose clean-only holdout is worse than M1b v3's 0.072 by more than 0.03 is not admitted whatever its In-the-Wild number says.
• The probe directory that produced it, so the runner can be pointed at it (--probe) and the constants can record its identity the way fusion_v2 records the M5 checkpoint.

Readouts. Identical to the M5 addendum, on identical row sets, CURRENT = the shipped A3 w0.2 + E computed from m1b_v3.

Decision rule (fixed now). R1 replaces the shipped rule only if all four hold, the same four as the M5 addendum: (1) In-the-Wild brief-cost minDCF improves by ≥ 0.01; (2) In-the-Wild miss-averse minDCF gets worse by no more than 0.01; (3) holdout brief-cost gets worse by no more than 0.05; (4) inner OOF gets worse by no more than 0.03. If R1 qualifies, it is put to Nathan with the three costs (a new probe in the runner, the constants' probe identity, the freeze clock); nothing ships without his word. If it does not qualify, the negative result is recorded here and the shipped rule stays.

Mechanics. uv run python scripts/fuse_sweep_m5.py --refit-m1b <name> [--refit-itw <path>] prints the table and the decision and writes outputs/fusion/sweep_refit_report.json; --write is refused in that mode (a refit's constants need the probe identity, built only on ratification). Self-check recorded below: with --refit-m1b m1b_v3 every R1 readout equals CURRENT and the decision is KEEP.

Expectation, stated in advance. A symmetric codec refit should help In-the-Wild false alarms (the wild-like regime λ̂ points at) and cost a little on the clean holdout; the clean-only floor above is what keeps that trade honest.

Outcome (13:05, oversight chat relaying the channel lane): not exercised. The codec search found no match (the 7.2 kHz wall is best reproduced by our own Kaiser low-pass; the closest MP3/AAC variant was 3% closer where the rule needed 20%), so the refit did not run and no column arrived; this rule stands as pre-declared and unexercised. The λ̂ = 0.947 headline in "Why now" was withdrawn by its lane (its dominant feature read the stopband); the final reading is indeterminate (0.51 with CI 0.12–0.64; 0.36 duration-matched; 0.48–0.57 across rate-consistent variants; 10 of 21 channel features put the test set outside the holdout–In-the-Wild span, so it is its own recording population rather than a blend of the two). The rule choice does not depend on it: fusion_v2 beats v1 on both the holdout and In-the-Wild.

Self-check result (12:58): --refit-m1b m1b_v3 on identical row sets (16,142 / 3,858 / 1,671 / 3,000): R1 equals CURRENT on every readout (inner 0.1351, holdout 0.0065 / 0.0087, ITW brief 0.228, averse 0.2385, P_FA/P_miss 1.3% / 12.2%), decision KEEP, nothing written under models/; outputs/fusion/sweep_refit_report.json.