The library

Rendered from docs/consults/2026-09-26_fusion-strategy_RESPONSE_verbatim.md; the markdown file is the record.

September 26, 2026

Consult: fusion, selection, false alarms and the "default answer" (Sat Sep 26, 2026, ~03:30 EDT; updated ~04:30 with In-the-Wild and band-limit findings; ~05:50 with the band-matched v3 numbers, M3 and the fusion v0 readout) Target: a frontier LLM, for strategic and technical opinion. Scope: everything except the M5 fine-tune recipe, which has its own consult: 2026-09-26_m5-extra-data-finetune_CONSULTATION.md. Blocking? Partly. The fusion and selection answers decide what we send for NSA's one-time draft review on Saturday afternoon. Copy below the line. Your role You are an applied researcher in audio deepfake detection and in detection-cost evaluation (ASVspoof, NIST SRE-style DCF). Push back hard on any wrong premise. We have about 26 hours left in a hackathon (final TSV due in about 23). Practical beats ideal. The problem Task. The NSA "HEARSAY" challenge at HackGT 13: output P(synthetic) per audio file, with 1.0 = synthetic. Test set. 1,671 unlabeled 16 kHz mono PCM WAVs, 3.0–13.6 s long (median 3.4 s), English. NSA says "approximately 70% of the files are real." Judging. 60% MinDCF. Points are (1 − ours)/(1 − best) × 60, where "best" is the best team's minDCF. 20% for diversity and depth of techniques. The brief itemizes: container/metadata, spectral, prosody, ENF, compression, speaker-embedding consistency, deep anti-spoofing, splice, and a bonus for "agentic orchestration" (choosing analyses per file). 20% for GitHub documentation "in your own words", including what did not work and explainability. Stated cost. A false alarm (real called synthetic) costs 4× a miss. With π_synth = 0.3, our normalized minDCF is 9.33·P_FA + P_miss. That is our selection metric; EER is logged alongside. The sponsor's scoring code contradicts the instructions. Their MinDCF code (shipped to us) is the ASVspoof5 Track 1 evaluation package, unmodified: Pspoof = 0.5, Cmiss = 1, Cfa = 4 compute_det_curve(bona, spoof) treats a higher score as bona fide The instructions say 1.0 = synthetic. Scored with that code as shipped, every good detector of ours gets minDCF 1.00. Scored with our scores flipped, the same detectors get 0.09–0.13. Our decision is to follow the instructions (1.0 = synthetic) and never flip. We have asked the sponsor; there's no answer yet. The instructions also say: "You will have to strategize and use some sort of Game Theory or Benchmark maxing to determine what the default answer to be if you cannot determine an answer for all of the files. This could be crucial!" Deliverables. The TSV (before 8 AM Sunday), a Docker image that runs inference offline on CPU, and a README. We also get one optional early review of a draft TSV against the test labels. Data and validation Training data (sponsor-provided plus public): DiffSSD spoof: 10 generators, 70k clips total, of which the training sample uses 1k per generator: LJ-voice TTS/vocoders: grad_tts, diffgan_tts, pro_diff, wavegrad2 multi-speaker: openvoicev2, unit_speech, xtts_v2, your_tts commercial: elevenlabs, playht LJ Speech (real, one speaker): 6k in the sample. LibriSpeech dev+test (real, about 80 speakers): 4k in the sample. Input path (identical for train and test): Decode through FFmpeg to 16 kHz. Trim silence (relative 35 dB). Per-input z-normalization. No tiling; training crops are drawn from the test-duration distribution; test-time uses the whole clip. Shortcuts found and neutralized: 7.2 kHz band wall. Every NSA test file is low-passed at about 7.2 kHz: the median energy share in 7.5–8 kHz is 1.6e-8, against 6e-4 to 6e-3 in LJ, LibriSpeech and DiffSSD. That band was a train-only cue. It's now removed by an identical Kaiser low-pass on train and test, in both the deep and engineered paths (since ~04:20). All deep numbers below predate this fix and are being re-measured. peak level (AUC 0.66 on its own) leading silence (LibriSpeech 0.37 s vs test 0.06 s) a tiling seam from padding short clips container format Folds (splits/nsa_folds.csv, groups: spoof by generator; bona fide by speaker, or chapter for LJ): Outer holdout: playht + wavegrad2 + 26 bona fide groups (1,858 real / 2,000 spoof). Never used for training or selection. Inner folds 0–4: leave-generator-out on the spoof side. Leakage: no exact file or PCM duplicates between the test set and LJ, the NSA LJ subset, or In-the-Wild. Current numbers Normalized minDCF at π = 0.3 unless stated. Holdout = outer holdout. Detector Inner-fold OOF minDCF Holdout minDCF Holdout EER Sponsor code as-is / flipped M1: frozen XLS-R 300M, layer 7 mean-pooled, logistic regression 0.250 0.146 2.5% 1.00 / 0.090 Handcrafted: 75 spectral + prosody features, LightGBM 0.614 0.255 4.3% 1.00 / 0.127 Compression forensics 0.931 0.900 20% 1.00 / 0.590 Container/metadata, ENF, splice 1.00 1.00 35–50% no signal (inputs are uniformly decoded) Stack: logistic on inner OOF of M1 + handcrafted 0.250 0.146 2.5% puts essentially all weight on M1 Mean of z-scored logits, M1 + handcrafted (no fitting) 0.258 0.066 1.3% – / 0.043 Detail: M1 per held-out generator: playht 0.032, wavegrad2 0.214 (the LJ-voice vocoder is the hard one). M1 per bona fide source: LJ 0.088 vs LibriSpeech 0.159. Handcrafted per bona fide source: LJ 0.10 vs LibriSpeech 0.32. Unfamiliar real voices draw more false alarms. M1 layer sweep, inner CV: L6 0.284, L7 0.250, L8 0.282, L12 0.281, L20+ ≥ 0.40. M1 on the NSA test set: Score distribution: 28% of files above 0.9 and 40% above 0.5 (NSA says about 30% synthetic). Preflight: pure silence scores 0.9996 and a synthetic music chord scores 1.0, i.e. confidently wrong on non-speech. The test set looks all-speech (median voiced fraction 0.8). Real-world stress test (In-the-Wild, evaluation only, never trained on) The sample is 2,000 real clips from 54 speakers plus 1,000 fakes, web-sourced celebrity audio. "Inner threshold" is the threshold that minimizes inner-fold out-of-fold DCF. M1 (NSA-only real) M1b (+ASVspoof 2019 LA real: 40 VCTK speakers, 5.1k clips, plus a 2.5k A01–A06 anchor) NSA holdout minDCF 0.146 0.079 (better on every slice; inside holdout noise) Inner-fold CV minDCF 0.250 0.248 In-the-Wild minDCF 0.374 0.405 In-the-Wild P_FA at the inner threshold 1.2% 5.7% Update ~05:40 (band-matched v3 and M3): Detector NSA holdout minDCF In-the-Wild minDCF In-the-Wild real P_FA Test share > 0.5 (prior ~30%) M1 v3 (band-matched) 0.159 0.380 0.7% 28.9% (v2 was 39.6%) M1b v3 (band-matched, +VCTK) 0.072 0.342 0.8% 26.8% M3 Spectra-AASIST (pretrained, frozen, band-matched) pending (~06:00) 0.065 0.0% (0 of 2,000) 26.5% (200-clip pilot) Handcrafted v4 (234 features, LightGBM) 0.170 1.00 (AUC 0.76) 0.65% 41% Fusion v0 readout (05:32; rules pre-declared, nothing fit on holdout, In-the-Wild or test). Inputs: M1b v3 and handcrafted v4. Each detector's logit is standardized on its inner out-of-fold rows. zmean = mean of standardized logits; rankmean = mean of per-detector ECDF ranks; stack_nonlj = logistic stacker fit on inner rows with LJ real speech excluded (one speaker on both sides of every fold), so its inner number is in-sample and not shown. Rule Inner OOF Holdout Holdout LibriSpeech real ITW minDCF ITW real P_FA ITW P_miss Test share > 0.5 M1b v3 alone 0.301 (folds include the extra real rows) 0.072 0.102 0.342 0.8% 27.8% 26.8% handcrafted v4 alone 0.434 0.170 0.217 1.00 0.65% 97% 56% zmean 0.259 0.018 0.022 0.387 0.15% 58.5% 39.6% rankmean 0.254 0.015 0.015 0.379 0.15% 59.5% 31.8% stack_nonlj (in-sample) 0.044 0.066 0.317 0.7% 31.6% 27.5% Reading: fusing the two cuts the holdout by 4× and In-the-Wild false alarms by 5×, but doubles In-the-Wild misses and pushes the test-set share of files called synthetic from 27% to 32–40%, against a 30% prior. The handcrafted column contributes almost no real-world detection (ITW P_miss 97%), so its In-the-Wild "gain" is a threshold effect. The stacker sits between. Spectra-AASIST's training data is undisclosed. Its card reports 1.46% EER on In-the-Wild, and it separates our NSA inner rows perfectly (AUC 1.0). So its In-the-Wild and inner-fold numbers may be in-sample. Band matching fixed the test-distribution shift but left the M1 In-the-Wild gap unchanged. New question: how should we weight a detector whose validation rows may be in its training data? Both models are 3–5× worse on real-world audio than on the NSA-domain holdout. The brief says the test's real audio may include smartphone, telephony and field recordings. Adding VCTK read speech did not help on In-the-Wild. An earlier consult predicted it would. Part of the gap may be the 7.2 kHz band mismatch (In-the-Wild isn't low-passed either); the re-measurement will show. What I want (ranked) For each question: your recommendation, a concrete procedure we can run in under 1 hour of CPU, and EV × effort. Selection under disagreement. The inner folds say the M1 + handcrafted mean is slightly worse than M1 (0.258 vs 0.250). The holdout says it is much better (0.066 vs 0.146). The logistic stacker trusts the inner OOF and zeroes out the handcrafted detector. What is the principled choice? What is the likely reason for the disagreement? The inner folds are leave-one-generator-out on 8 generators. The holdout has only 2 generators. The handcrafted inner OOF is weak on grad_tts and pro_diff (1.0). Would rank averaging, per-fold weight stability, a constrained stacker (non-negative weights, a shrink toward equal weights) or nested CV of the fusion rule resolve it without peeking at the holdout? Is it acceptable to use the one-time draft review to choose between two candidates? Which two should we send? With the fusion v0 table above: the unfitted rules (zmean, rankmean) win the holdout and In-the-Wild false alarms by a wide margin but raise In-the-Wild misses from 28% to 59% and the test share above 0.5 from 27% to 32–40%. The stacker fit on non-LJ rows is in between on everything. Which of the four do we ship at 14:00, and is a rising test share a warning sign (fusion promoting borderline real files) or noise? Weighting a pretrained detector whose validation rows may be in-sample. M3 (Spectra-AASIST) separates our inner rows perfectly (AUC 1.0) and reports In-the-Wild real P_FA 0.0%, but its training data is undisclosed and LJ, LibriSpeech, DiffSSD and In-the-Wild are all public. A stacker fit on inner rows will give it all the weight. How should its weight be set: holdout only, rank mean with equal weight, a cap, or excluded from stacking and used only as a gate? The "default answer" / game-theory hint. With a rank-based MinDCF at π = 0.3 and C_FA = 4, where should files we "cannot determine" sit in the ranking? Examples: decode failures, non-speech, detector disagreement, low confidence. Is there a smarter policy than the prior? Push uncertain files to the real end, because false alarms dominate. Break ties deliberately. Account for the minDCF threshold sweep. How does the answer change if NSA actually runs their shipped code unmodified (a higher score treated as bona fide, Pspoof = 0.5)? Is there any hedge that is safe under both directions, or is the draft review the only real protection? Channel robustness: false alarms on real-world real speech. This is now the top question. Our real training data is one LJ speaker, about 80 LibriSpeech readers, and optionally 40 VCTK speakers, all clean read speech. In-the-Wild shows 3–5× worse minDCF, and more read-speech speakers didn't help. The test's real audio may include smartphone, telephony and field recordings. What is the cheapest effective lever in the next ~20 hours, using In-the-Wild real-speech P_FA as the proxy? channel augmentation on both classes: telephony band-limit, codecs, RIR, noise, RawBoost convolutive score normalization test-statistics standardization or CORAL on the unlabeled test embeddings which bona fide corpora actually add channel variety (not just more read speech) center-loss or one-class objectives (another consult warned these are contraindicated with narrow bona fide data) What is safe to do with the unlabeled test set? We have ruled out pseudo-labeling. Is checking whether test real clips are LJ-like (nearest-neighbor to LJ embeddings) useful or a trap? Explainability and the diversity score. Our engineered detectors carry no signal because every test file is identical in format. Should we: keep them in the orchestrator anyway (to cover the rubric), replace them with techniques that work on decoded 16 kHz audio (e.g. speaker-embedding drift within a clip, breath/pause statistics, formant or phase features), or just document them honestly as "no effect"? What would a judge value for "agentic orchestration" beyond rule-based routing? The non-speech failure. Silence and music score about 1.0. What is the right gate (a VAD threshold on voiced fraction, a speech/music classifier), and what score should gated files get, given question 2? Time allocation. About 26 h remain (23 to the final TSV), and one person owns the deep models, fusion, TSV and the draft review; a fine-tune, a pretrained-detector scoring run and the engineered detectors run in parallel lanes; Docker is unowned. What would you cut, and what single change do you expect to move holdout minDCF the most? Hard lines No training or selection on the outer holdout. Never train on In-the-Wild. No pseudo-labeling of the test set. Output 1.0 = synthetic (instructions). Offline CPU inference in Docker, no hosted APIs. Training on external datasets is allowed by the brief. Output format A one-line verdict on our current direction. A ranked actions table: action | expected minDCF impact | effort | risk | do / maybe / skip. For each of the six questions: a recommendation with concrete numbers and one runnable procedure. The draft-review plan: which TSV(s) to send, what to learn from the returned score, and how to act on it. Failure modes we haven't named, especially around the 4× false-alarm cost and the sponsor-code direction.

Executive summary

Scope

Fusion rule selection, detector weighting for a possibly-contaminated pretrained model (M3), score-direction and default-answer policy under sponsor-code ambiguity, channel-robustness triage, non-speech gating, diversity/orchestration tactics, draft-review payload choice, and lane allocation for the NSA HEARSAY audio deepfake detector ahead of the Saturday one-shot draft review and the Sunday 08:00 TSV deadline. Excludes the M5 fine-tune recipe.

Conclusion

Ship an M1b-dominant ranking, not the 50/50 rank/z fusion: the unfitted rules buy ~0.03 holdout DCF (≈6 holdout files) while doubling In-the-Wild misses from 28% to 59%. stack_nonlj (holdout 0.044, ITW 0.317) is the minimax-regret choice; M1b v3 alone is the defensible fallback and the α=0 endpoint of the same sweep. Spend the single draft shot on the actual shipped artifact to read score direction — worth ~57 points versus ~2 for fusion discrimination. Pin indeterminate and non-speech files to the MINIMUM submitted score and never flip that block: it stays optimal for block spoof rate $q\in[0.097,0.80]$ under both score conventions. Admit M3 only as a capped vote or FA-suppressing gate. Assign Docker an owner now.

Both experts independently rejected the equal-weight rank/z fusion on the same evidence — the In-the-Wild miss doubling is a ~20σ effect while the holdout gap is ~6 files — and converged on pushing indeterminate files to the real end, VAD-gating non-speech, treating M3 as a gate rather than a stacked feature, and using the draft shot to read score direction. They split on the shipped rule (stack_nonlj vs M1b alone, which are endpoints of one α-sweep) and on whether a direction-robust hedge exists; Gemini missed that the shipped ASVspoof code also inverts the cost semantics. Score direction, M3 provenance and the test domain mixture remain unmeasured.

Sensitivities

• The stack-versus-rankmean call reverses if the test set turns out to be substantially wild-channel — the crossover sits at an estimated wild-domain weight of about 0.32.

• If the draft review returns a value near 1.00, the entire determinate ranking must be flipped and the rule re-selected under the miss-averse cost, which favours the M1b-dominant family even more strongly.

• M3's role is contingent on its pending holdout number: a capped vote only below 0.05, gate-only in between, and full exclusion above 0.10.

• The non-speech gate's value depends entirely on how many test files actually trip it — zero files means 30 minutes lost, twenty files means roughly 10 points gained.

Final Verified Code

Claude's scenario-algebra block encodes the two coupled ambiguities — score polarity and the cost-semantics inversion in the shipped ASVspoof5 script — solves the break-even indeterminate-block spoof rate $q^{*}$ in each scenario with brentq, re-ranks the fusion rules under both cost weightings, and emits the draft-review decoding table. It is the only artifact that substantiates the direction-robust hedge.

Show code

import numpy as np, pandas as pd, matplotlib
matplotlib.use("Agg"); import matplotlib.pyplot as plt

# ===== Scenario algebra: TWO independent ambiguities =====
# Brief:  high score = synthetic ; expensive error = real called synthetic (FA), 4x, pi=0.3
#         DCF_S1 = P_miss + 9.333*P_FA        (normalise by min(pi*Cmiss,(1-pi)*Cfa)=0.3)
# Shipped ASVspoof5 code: high score = bona fide ; Pspoof=.5, Cmiss=1 (bona fide REJECTED), Cfa=4 (spoof ACCEPTED)
#         unnormalised = Cmiss*(1-pi_s)*Pmiss_bona + Cfa*pi_s*Pfa_spoof = .5*Pmiss + 2*Pfa ; /min(.5,2)=.5
#         DCF_S3 = Pmiss_bona(=real rejected) + 4*Pfa_spoof(=spoof accepted)
print("S1 weights: P_miss(spoof called real) x1 , P_FA(real called synth) x9.333  -> expensive claim = 'synthetic' = HIGH score")
print("S3 weights: P_miss_bona(real rejected) x1 , P_fa_spoof(spoof accepted) x4  -> expensive claim = 'bona fide' = HIGH score")

pi_true=0.30
def block_cost(q, where, scenario):
    """cost added per unit block mass u; where in {'min','max'} of the SUBMITTED score"""
    if scenario=='S1':   # high = synthetic
        return q/pi_true if where=='min' else 9.3333*(1-q)/(1-pi_true)
    else:                # S3: high = bona fide, we flipped determinate scores
        return (1-q)/(1-pi_true) if where=='min' else 4.0*q/pi_true
rows=[]
for q in [0.05,0.10,0.20,0.30,0.50,0.70,0.80,0.90]:
    r={'q (spoof rate in block)':q}
    for sc in ['S1','S3']:
        cmin,cmax=block_cost(q,'min',sc),block_cost(q,'max',sc)
        r[f'{sc} cost@MIN']=round(cmin,3); r[f'{sc} cost@MAX']=round(cmax,3); r[f'{sc} best']='MIN' if cmin<=cmax else 'MAX'
    rows.append(r)
T=pd.DataFrame(rows); print("\n", T.to_string(index=False))
# break-evens
from scipy.optimize import brentq
q1=brentq(lambda q: block_cost(q,'min','S1')-block_cost(q,'max','S1'),1e-6,1-1e-6)
q3=brentq(lambda q: block_cost(q,'min','S3')-block_cost(q,'max','S3'),1e-6,1-1e-6)
print("\nbreak-even q*: S1=%.3f (put at MIN if q<q*)   S3=%.4f (put at MIN if q>q*)"%(q1,q3))
print(">>> DIRECTION-ROBUST WINDOW for 'indeterminate block at the MINIMUM submitted score': q in [%.3f, %.3f]"%(q3,q1))

# ===== rule ordering under both cost weightings (ITW, at inner threshold) =====
itw={'M1b v3':(0.008,0.278),'handcrafted v4':(0.0065,0.97),'zmean':(0.0015,0.585),
     'rankmean':(0.0015,0.595),'stack_nonlj':(0.007,0.316)}
out=[]
for k,(pfa,pm) in itw.items():
    out.append(dict(rule=k,P_FA=pfa,P_miss=pm,
                    DCF_S1_at_thr=round(pm+9.3333*pfa,3),
                    DCF_S3_at_thr=round(pm+4.0*pfa,3)))
D=pd.DataFrame(out).sort_values('DCF_S3_at_thr'); print("\nITW cost at the inner threshold under both cost structures:\n",D.to_string(index=False))
print("\n(reported ITW *minDCF* under S1 weights: M1b .342 | zmean .387 | rankmean .379 | stack .317)")

# ===== figure: scenario matrix =====
fig,ax=plt.subplots(figsize=(9,4.2)); ax.axis('off')
cell=[["submit 1.0=synthetic\n(current plan)","~0.02-0.15  -> 53-60 pts","1.00 (ceiling) -> 0 pts"],
      ["submit 1.0=bona fide\n(flip)","1.00 (ceiling) -> 0 pts","~0.09-0.13 -> 54-58 pts"]]
tb=ax.table(cellText=[[c for c in r] for r in cell],
            colLabels=["our TSV","S1: graded per the brief\n(high=synth, FA x4, pi=.3)","S2/S3: graded by shipped code\n(high=bona fide, spoof-accept x4)"],
            loc='center',cellLoc='center'); tb.auto_set_font_size(False); tb.set_fontsize(9); tb.scale(1,2.2)
ax.set_title("One binary unknown decides ~57 of 100 points. The draft submission is the only instrument that reads it.",fontsize=10)
plt.tight_layout(); plt.savefig("fig4_scenario_matrix.png",dpi=140)

# ===== draft-review decoding table =====
dec=[("~0.00 - 0.20","direction = ours (1.0=synthetic) AND test behaves like the holdout","ship the same rule; do not flip; spend remaining time on README/diversity"),
     ("~0.20 - 0.45","direction = ours, but test is wild-like / out-of-family generators","switch to the most FA-robust candidate (stack/alpha-fusion + wild-real refit); do not flip"),
     ("~0.45 - 0.90","ambiguous: severe domain failure OR partial mis-read","do NOT flip (a flipped good detector lands at the 1.00 ceiling, not 0.6); fall back to M1b v3 alone + aggressive default block"),
     ("0.95 - 1.00","score direction is the shipped one (1.00 is exactly the trivial-system ceiling)","FLIP the final TSV (submit 1.0 = bona fide) AND keep the indeterminate block pinned to the MINIMUM value; re-rank for a miss-averse (4x) cost")]
DR=pd.DataFrame(dec,columns=["returned minDCF","inference","action"]); DR.to_csv("draft_review_decoder.csv",index=False)
print("\n",DR.to_string(index=False))

Expert Summaries (2)

Claude

Approach:

Reframed the problem as a points-allocation question, derived the points-per-DCF slope and per-file error costs, then used executed decision-theoretic arithmetic (break-even $q^{*}$, minimax regret, $\lambda$ crossover, Poisson intervals on holdout false-alarm counts) to rank every open decision by expected value against effort.

Conclusion:

Ship stack_nonlj (minimax regret 0.029 vs rankmean's 0.062), send the real shipped artifact as the draft to read score direction, and pin indeterminate files to the minimum submitted score — the one hedge that is optimal under both score conventions.

Strengths:

• Identified a second, coupled ambiguity nobody had named: the shipped ASVspoof script inverts the cost semantics as well as the polarity, so the expensive claim sits at the HIGH end of the submitted file under both conventions

• Converted every DCF gap into file counts (1 holdout FA = 0.0050 DCF; rankmean-vs-stack = ~5.8 files) and showed the holdout is at its resolution floor while the ITW $P_{miss}$ gap is ~20σ

• Caught that the stated inner-vs-holdout contradiction is stale — on the v3/v4 rows inner OOF and holdout now agree that fusing beats M1b alone

• Corrected the VCTK premise: post-band-fix, M1b v3 improved ITW 0.380→0.342 and holdout 0.159→0.072

• Correctly identified that test share above 0.5 is a calibration statistic that cannot change $\mathrm{minDCF}$

• Produced executed code with break-even solutions and a draft-review decoding table

Weaknesses:

• Several headline arithmetic claims (points slope, per-file costs) rest on an assumed best-team $\mathrm{minDCF}=0.05$ that is unknowable

• Cited ASVspoof 5 published cost constants that did not verify against the source, and quoted UR-AIR augmentation figures that graded as incorrect

• Ten-rule A/B menu risks selection-criterion overfitting on the ITW proxy despite the pre-declared objective

• stack_nonlj's inner number is in-sample, so the recommendation leans on two estimates rather than three

Unique:

The cost-semantics inversion inside the sponsor's own script, and the resulting proof that a minimum-pinned indeterminate block is direction-robust for $q\in[0.097,0.80]$.

Gemini

Approach:

Argued from the fold-composition asymmetry (8 leave-one-generator-out inner folds vs a 2-generator holdout) that the inner folds and In-the-Wild should overrule the holdout, and recommended the simplest defensible artifact plus waveform augmentation.

Conclusion:

Ship M1b v3 alone with no fusion, send that same clean baseline as the draft to read score direction, gate non-speech and indeterminate files to the real extreme, and spend GPU time on RawBoost augmentation rather than new corpora.

Strengths:

• Correct diagnosis of why the holdout flatters the handcrafted detector: its inner-OOF failures (grad_tts, pro_diff) are absent from the 2-generator holdout

• Clean mechanistic account of why standardize-and-average flattens the M1b distribution and converts confident spoof scores into misses

• Right call on M3: perfect inner separation makes any fitted stacker overfit, so use it as a confidence gate

• Identified the nearest-neighbour-to-LJ diagnostic as a trap and recommended honest documentation of the dead engineered detectors

Weaknesses:

• Claimed no direction-agnostic hedge exists — this is wrong because it models only the polarity flip and not the cost-semantics inversion in the shipped script

• Repeated the stale 'VCTK read speech did not help' premise that the post-band-fix v3 numbers contradict

• Top-ranked action (RawBoost on the M5 fine-tune, 5 h GPU) reaches into the explicitly out-of-scope fine-tune recipe

• Supporting code was a four-line default-cost check that does not substantiate the threshold-sweep claim it was cited for

• No per-file cost arithmetic, no treatment of the non-speech gate's magnitude, and no time-lane or Docker triage

Unique:

The clearest short statement of the fold-composition mechanism and of the M3 confidence-gate procedure; a useful contrarian anchor at the $\alpha=0$ endpoint.

Assumptions and Limitations

Assumptions

• Best-team $\mathrm{minDCF}$ is assumed near 0.05 for the points-per-DCF slope (~63 points per unit DCF); the slope moves if the true best is much lower or higher.

• Test set composition is assumed to match the stated prior: ~1,170 real and ~501 spoof out of 1,671, which makes one false alarm exactly 4× one miss.

• Holdout composition is 1,858 real and 2,000 spoof, giving 0.0050 DCF per false alarm and 0.00050 DCF per miss.

• The brief's cost structure is $\mathrm{minDCF}=P_{miss}+9.333P_{FA}$ at $\pi_{synth}=0.3$, $C_{fa}=4$, $C_{miss}=1$.

• The shipped ASVspoof5 script is unmodified with $P_{spoof}=0.5$, $C_{miss}=1$, $C_{fa}=4$, and treats a higher score as bona fide, so the $4\times$ weight falls on spoof-accepted.

• Indeterminate files are assumed to have internal spoof rate $q\approx\pi_{synth}=0.30$, with no evidence that any category exceeds $q=0.80$.

• The fusion v0 table values (holdout, ITW, inner OOF, test share) are taken as reported and were computed without fitting on holdout, ITW or test.

• All reported v3 numbers are post-band-match; the earlier VCTK null result is treated as confounded by the 7.2 kHz shortcut.

• $\mathrm{minDCF}$ is invariant to any monotone transform of the scores, so only the ranking matters.

• Only one draft TSV can be submitted, with no resubmission, and the returned feedback is assumed to be at least one scalar.

Limitations

• M3's training-data provenance is undisclosed; neither contamination nor cleanliness was established, and its holdout number was still pending at the time of analysis.

• The wild-domain weight $\lambda$ of the test set is unmeasured, and the stack-versus-rankmean recommendation flips at $\lambda=0.319$.

• The sponsor's actual grading polarity and cost constants remain unconfirmed; both scenarios are live.

• stack_nonlj's inner-fold number is in-sample and unusable, so its selection rests on only two honest estimates.

• The holdout cannot resolve differences smaller than about 5–6 false alarms, so its rule ordering is near the noise floor.

• No expert could quantify the expected gain from adding wild bona fide data in a frozen-embedding setup; published figures come from full in-domain training.

• The codec hypothesis for the 7.2 kHz edge is untested.

• Several externally cited figures (RawBoost's mechanism, VoxCeleb access friction, the SAMO clustering result, In-the-Wild degradation magnitudes) did not verify against sources and are carried as reported-but-unverified; one expert's ASVspoof 5 constant and UR-AIR augmentation figures graded as incorrect and were dropped.

Figures (4)

Claude (4 figures)

Nomenclature (18 symbols)

Narrative

Expert Comparison

The two experts agree on more than they disagree on. Both reject the unfitted 50/50 rank and z means; both diagnose the same mechanism (the handcrafted detector's inner-OOF failures on grad_tts and pro_diff are absent from the 2-generator holdout, so the holdout is a favourable draw for it); both send the indeterminate and non-speech files to the real end; both refuse to let a fitted stacker see M3; both call the rising test share a threshold artifact rather than a signal; both flag the nearest-neighbour-to-LJ diagnostic as a trap; both use the single draft shot to read score direction rather than to compare candidates.

The disagreements are narrower than they look. stack\_nonlj and M1b v3 alone are not opposing philosophies — they are two points on the same weight continuum, with the stacker placing modest non-zero weight on the handcrafted column and M1b alone placing zero. The substantive splits are (i) whether a direction-robust hedge exists, where Gemini is demonstrably wrong, (ii) what to put in the draft TSV, and (iii) whether wild real data or waveform augmentation is the cheaper channel-robustness lever.

Expert Divergence

Divergence 1 — the direction-robust hedge. Gemini asserts flatly that "there is no direction-agnostic hedge" because the $\mathrm{minDCF}$ threshold sweep will always group an intermediate score with the wrong class under one of the two conventions. That argument is correct for the determinate ranking and wrong for the extremes. Gemini modelled only the polarity flip. Claude modelled both ambiguities and showed they interact: the shipped ASVspoof package treats a higher score as bona fide, so with $P_{spoof}=0.5,\ C_{miss}=1,\ C_{fa}=4$ the $4\times$ weight lands on spoof accepted, not on real called synthetic. The expensive claim therefore sits at the HIGH end of the submitted file under both conventions, and pinning the indeterminate block to the minimum is optimal in both. Claude's executed solve gives break-even $q^{*}=0.800$ under the brief (pin to minimum if $q<q^{*}$) and $q^{*}=0.0968$ under the shipped code (pin to minimum if $q>q^{*}$), a robust window of $q\in[0.097,\,0.800]$ with the indeterminate block sitting at $q\approx\pi_{synth}=0.30$, roughly dead centre. Claude's position is adopted. Note that Gemini's supporting code was a four-line comparison of all-real (0.3) versus all-spoof (2.8) default cost — it establishes the FA asymmetry but does not substantiate the no-hedge claim it was offered for.

Divergence 2 — which rule to ship. Both agree the equal-weight rules are out. Claude prefers stack\_nonlj on three grounds: minimax regret across the holdout and ITW axes (0.029 vs rankmean's 0.062), the fact that the axis it wins on is far better powered, and its survival under the inverted $4\times$-miss cost structure. Gemini prefers M1b v3 alone on simplicity and on distrust of anything fitted. The resolution is that these are the $\alpha=0$ and small-$\alpha$ ends of a one-parameter rank blend, and the cheapest way to settle it is the $\alpha$-sweep both frameworks imply — run it this morning, report all metrics, and pick by a pre-declared objective. Ship stack\_nonlj if the sweep confirms a non-zero optimum; ship M1b v3 if it does not. Gemini's caution has independent support: the SZU-AFS team reported that combining classifiers sometimes lands between the individuals with no improvement, and that using only the best classifier was sufficient [S17].

Divergence 3 — the VCTK premise. Gemini repeats the team's stated conclusion that adding VCTK read speech did not help In-the-Wild. Claude traces this to a pre-band-fix measurement (0.374 vs 0.405) confounded by the 7.2 kHz shortcut, and points at the team's own post-band-match v3 numbers: M1 v3 ITW 0.380 → M1b v3 ITW 0.342, holdout 0.159 → 0.072. VCTK helped on both axes once the confound was removed. Claude's correction is adopted, and it materially changes the channel-robustness recommendation: the speaker/channel-diversity hypothesis should not be retired.

Divergence 4 — scope. Gemini's top-ranked action is RawBoost augmentation on the M5 fine-tune with 5 h of GPU. The case file explicitly excludes the fine-tune recipe from this consult. The augmentation principle is retained for the frozen-embedding path; the fine-tune routing is out of scope and is not carried forward as a recommendation here.

Technical Detail

The scoreboard arithmetic that frames everything

Points are $(1-\mathrm{ours})/(1-\mathrm{best})\times 60$. If the best team lands near 0.05, the slope is

\frac{d(\text{points})}{d(\mathrm{minDCF})} = -\frac{60}{1-0.05} \approx -63\ \text{points per unit DCF},

so the entire spread between an excellent 0.05 submission and a mediocre 0.15 one is about 6 points out of 100. The rankmean-versus-stack decision is worth roughly 2 points on the holdout axis and 4 on the ITW axis. Getting the score direction wrong costs essentially all 60 minDCF points, because a good detector submitted with the wrong polarity lands at the trivial-system ceiling of 1.00 — which is exactly what the team already observed when running their own scores through the shipped package unmodified. That asymmetry — a ~2-point question versus a ~57-point question versus a 40-point README-and-diversity question — is the single most important reframing either expert produced, and it should drive lane allocation for the rest of the day.

On the test set ($\approx$1,170 real / 501 spoof, matching the 0.3 prior), one false alarm costs 0.0080 DCF and one miss 0.0020 DCF — exactly the stated 4:1. On the holdout (1,858 real / 2,000 spoof), one false alarm is 0.0050 DCF and one miss 0.00050 DCF. These per-file conversions are what make the rest of the analysis tractable.

Item 1 — Selection under disagreement

The disagreement as originally framed (inner OOF 0.250 vs 0.258; holdout 0.146 vs 0.066) is stale: those are pre-band-fix M1 plus handcrafted v1 numbers on a different row set. On the v3/v4 rows in the fusion v0 table, inner OOF orders rankmean 0.254 < zmean 0.259 < M1b v3 0.301 and the holdout orders 0.015 < 0.018 < 0.072. Inner and holdout now agree that fusing beats M1b alone. The live disagreement is between the NSA domain (inner + holdout, both say fuse) and In-the-Wild (says don't). That is a domain-weighting question, not a statistical-selection question, and no resampling machinery answers it.

The mechanism is separable. Fusion with handcrafted v4 (a) suppresses false alarms by requiring agreement on real speech — holdout LibriSpeech-real 0.102 → 0.015, ITW real $P_{FA}$ 0.8% → 0.15% — and (b) softens spoof scores, ITW $P_{miss}$ 27.8% → 59.5%. Effect (a) replicates across domains; effect (b) is pure damage on out-of-family spoofs, because handcrafted v4 has ITW $\mathrm{AUC}=0.76$ but $P_{miss}=97\%$ — ranking signal with no usable operating point. Under $W=9.333$ the ITW trade is a false-alarm gain of $9.333\times0.0065=0.061$ against a miss loss of 0.317, net −0.256, which is why ITW minDCF moves 0.342 → 0.379. Gemini's fold-composition explanation completes the picture: handcrafted v4's inner-OOF failures are grad_tts and pro_diff (both 1.00), and neither generator is in the 2-generator holdout, so the holdout is a favourable draw for it.

The holdout is also at its resolution floor. The rankmean-versus-stack gap of 0.029 DCF is about 5.8 false alarms or 58 misses; the rankmean-versus-zmean gap of 0.003 is 0.6 files. At three observed false alarms the 95% Poisson interval spans roughly 0.6 to 8.8 files. Meanwhile the ITW $P_{miss}$ gap is ~317 files against a binomial SD near 16. This is precisely the regime Cawley and Talbot warn about: selection-criterion over-fitting produces effects comparable in size to the true differences between candidates [S19], and their prescribed remedy for a high-variance selection criterion is a larger validation sample [S19] — which In-the-Wild is.

Run the ten-minute decomposition first: for each rule, print the argmin threshold on the holdout and the raw $(\#FA, \#miss)$ counts there. This is reading a breakdown of a number already computed, not selecting on the holdout. If rankmean's advantage is mostly false-alarm-count-driven, the two are tied on the holdout and the choice falls to ITW.

On the proposed machinery: rank averaging is already a candidate rather than a fix; per-fold weight stability adds nothing when the instability source is already known; nested CV is correct in principle but its outer loop still contains only the eight in-family generators, so it buys a better estimate of the wrong quantity; the constrained stacker is worth doing, but as a one-parameter $\alpha$-sweep that costs twenty minutes rather than a full nested framework.

Item 2 — Draft-review usage

Using the one shot to discriminate between fusion candidates is unsound: a single scalar reports the test minDCF of the one ranking you sent and says nothing about the counterfactual, and the question is worth 2–4 points against 12–30 points in expectation for the direction read. Both experts agree the shot goes to direction.

They split on payload. Gemini wants M1b v3 alone as a clean, well-understood baseline; Claude wants the actual shipped artifact, complete with gating, the pinned block, tie-jitter and exact TSV formatting. Claude's argument wins on breadth of return: one number then yields direction, the true test minDCF of the ranking you will actually ship, a read on how badly the holdout over-states performance, and format/parse validation — a malformed TSV is a silent zero with no second chance. The direction read survives either payload because the two hypotheses sit an order of magnitude apart, so there is no interpretive cost to sending the real artifact.

Decoding table, adopted from Claude's executed decoder:

If EER is returned instead, roughly 2–5% means our direction and roughly 95–98% means theirs. Ask explicitly for $P_{FA}$, $P_{miss}$ and EER alongside minDCF — it costs nothing and the decomposition would settle the holdout-resolution question for free. Pre-compute both polarity TSVs before 14:00 so the post-feedback action is a file copy, not a pipeline run.

One tempting play to reject: a two-level benchmark-maxing TSV (subset at 1.0, rest at 0.0) intended to solve the returned scalar for a label count. Claude evaluated it numerically and found the normalized DCF cap at 1.00 saturates the return for essentially every realistic subset composition — not decodable, and it burns the direction read.

Item 3 — The fusion rule at 14:00

On the two honest estimates, zmean (0.018, 0.387) is dominated by rankmean (0.015, 0.379) on both axes and should be dropped; M1b v3 (0.072, 0.342) is dominated by stack_nonlj (0.044, 0.317) on both axes, leaving a Pareto frontier of {rankmean, stack_nonlj} with M1b retained as the panic fallback. Between the frontier pair, stack_nonlj wins on minimax regret (0.029 vs 0.062), wins on the better-powered axis, and is the only one that survives the inverted cost structure — rankmean's entire edge is an FA-minimisation strategy tuned to $W=9.333$, and if the grader weights spoof-accepted at $4\times$, a 59.5% ITW $P_{miss}$ becomes the expensive error. Claude's re-ranking under the inverted weights puts M1b at 0.310 and stack at 0.344 against rankmean at 0.601 and zmean at 0.591, so the M1b-dominant family is robust in both worlds and the equal-weight family is not.

The pre-declared swap rule: estimate $\hat\lambda$, the wild-domain weight of the test set, from label-free channel statistics (95% spectral rolloff, long-term tilt, noise-floor level and stationarity, a reverberation proxy), train a tiny domain classifier to separate holdout-real from ITW-real, and average its predicted P(wild) over the 1,671 test files. The crossover is $\lambda=0.319$: below it rankmean wins, above it stack wins. Claude's falsifiable prior is that a uniform 7.2 kHz wall on every file suggests a single curated re-encode pipeline rather than genuine heterogeneous field audio — real telephony would show a 3.4 kHz mode. This is the cheapest decision-relevant measurement left.

Is the rising test share a warning sign? No — it is a category error. $\mathrm{minDCF}$ is a minimum over a threshold sweep and depends only on the ranking. Any monotone transform — the 0.5 cut, z-scoring, T-norm, Platt scaling, test-ECDF ranking — changes the share above 0.5 and changes minDCF by exactly zero. Gemini reached the same verdict by a different route (threshold flattening), but Claude's framing is the operative one: score normalisation to "fix" the share is worth exactly zero points, and the ITW $P_{FA}$-at-inner-threshold figures are likewise calibration statistics the grader will never compute. The relevant Bayes threshold, if one is ever needed, is $W/(1+W)=0.903$ under the brief and 0.800 under the shipped code — not 0.5.

The share retains one legitimate diagnostic use: fit the observed test-score CDF as a two-component mixture of the inner-OOF class-conditional CDFs and estimate $\hat\pi$. A value near 0.30 is consistent with the prior; near 0.45 indicates the real-side distribution has shifted upward, which is genuine FA inflation. $\hat\pi$ is itself biased under class-conditional shift, so treat it as a smoke alarm, not a selection metric.

Rules worth A/B-ing this morning, all cheap in the existing script: the $\alpha$-sweep in rank domain $(1-\alpha)r_{M1b}+\alpha r_{HC}$ over $\alpha\in\{0,0.1,\ldots,0.5\}$ (highest EV — the FA benefit saturates fast while the $P_{miss}$ damage grows roughly linearly, so expect an optimum near $\alpha\approx0.2\text{–}0.3$); the same sweep in z domain; the min-rule and max-rule as the maximally FA-averse and miss-averse bounds; a cascade veto that uses $r_{M1b}$ everywhere except an ambiguous mid-band where it averages in the handcrafted rank; median-of-three including M3; a capped weighted mean; a non-negative stacker shrunk toward equal weights; and a tie-jitter wrapper. Report every rule on inner OOF, holdout, per-real-source, ITW under both cost weightings, and $\hat\pi$. Pre-declare the selection objective before looking, cap the menu, and do not iterate — selection over ten rules on a proxy set is exactly the over-fitting risk Cawley and Talbot quantify [S19].

Item 4 — Weighting M3 under possible contamination

The structural fact both experts converge on: LJ Speech, LibriSpeech, DiffSSD and In-the-Wild are all public, as are the holdout's two generators and 26 bona fide groups. There is therefore no guaranteed-clean evaluation set for M3 anywhere in the team's possession — "calibrate on the holdout only" moves the contamination one level up rather than containing it. The only guaranteed-unseen data is the 1,671 unlabelled test files.

Note also that near-perfect separation on a development split is weak evidence of generalisation even with strictly disjoint protocols: in-domain AASIST scores 0.61% EER and 0.0160 minDCF on SpoofCeleb's validation split but 2.37% and 0.0328 on its evaluation split [S5]. So $\mathrm{AUC}=1.0$ on eight generators is not by itself proof of leakage — Gemini's word "guarantees" is too strong — but combined with undisclosed training data and universally public evaluation sets, it is not evidence of strength either. Treat it as uninformative.

The policy follows: never let a fitted stacker see M3. Admit it only as a capped vote in an unfitted rank rule (median-of-three, or weight 0.25) and only if the 06:00 holdout number lands below 0.05; exclude entirely if holdout exceeds 0.10, which would mean neither leaked nor strong. Gemini's confidence-gate variant is the same idea implemented as routing: defer to M3 only on extreme scores. Claude adds an important refinement — restrict M3 to FA suppression, damping scores where M1b says synthetic but M3 says strongly bona fide, and never allowing it to promote a file toward synthetic. The rationale is that M3's most useful claim (0 of 2,000 ITW false alarms) is on the real side, and memorising real speech is far less dangerous than memorising spoof generators.

Three cheap leakage probes, roughly an hour total. First and best: perturbation sensitivity — take 500 inner clips, apply label-preserving perturbations (low-bitrate MP3 round-trip, +20 dB SNR noise, ±2% time-stretch, one-sample circular shift), re-score with both M3 and M1b, and compare $\Delta\mathrm{AUC}$; a model that memorised specific files degrades far more under tiny perturbations than one that learned generalisable artifacts. Second: Spearman $\rho(r_{M1b},r_{M3})$ on the holdout versus on the test set — a large drop is direct evidence that holdout agreement is memorisation-driven, and it uses no labels. Third: score the MLAAD spoof already on disk; near-perfect on classic public sets but mediocre on MLAAD is consistent with memorisation. Spend ten minutes grepping the model card and repo for a training-data list, then stop.

Item 5 — Default answer and game theory

For a block of mass $u$ with internal spoof rate $q$, placing it at the bottom of the submitted ranking costs $uq/\pi_{synth}$ and at the top costs $Wu(1-q)/(1-\pi_{synth})$, giving

q^{*}=\frac{W/0.7}{1/0.3+W/0.7}.

Under the brief, $W=9.333$ gives $q^{*}=0.800$: pin to the minimum if $q<0.800$. Under the shipped code's inverted semantics, $q^{*}=0.0968$: pin to the minimum if $q>0.0968$. The indeterminate files sit at $q\approx0.30$, comfortably inside $[0.097,0.800]$ with roughly 3× margin on both sides. So "push uncertain files to the real end" is genuinely optimal, not a vibe — and it is the only hedge that holds under both score conventions. Gemini's independent all-real (0.3) versus all-spoof (2.8) check confirms the direction of the asymmetry under the brief.

Implementation is about placement, not value. Map determinate scores into $[0.001,1.000]$; put the indeterminate block in $[0.000,0.001)$, strictly below everything. Do not emit a constant — order within the block by whatever weak signal remains (raw M1b score, voiced fraction, duration) and use a deterministic hash jitter for genuinely signal-free files. Because minDCF is a minimum over available thresholds, distinct values offer the evaluator a superset of cut points and can only weakly improve the score; the expected gain is on the order of 0.001–0.002 DCF but is never negative. Placing the block in the middle is weakly dominated: it constrains the sweep without buying anything.

Category policy: decode failures and non-speech go to the bottom of the block. High-disagreement files should not be collapsed into the block — they carry noisy but real ranking information, so compress them toward the low end of the determinate range instead, which the $\alpha$-fusion and the cascade veto already do. Low-confidence files near the middle stay where they are; do not manufacture certainty.

If the shipped code is used, the block policy is unchanged but the determinate ranking must be flipped, and rule selection should shift to the miss-averse criterion. No hedge protects the ranking of the determinate files — a ranking and its reverse cannot both separate — so the draft review really is the only protection there. That is the correct, narrower version of Gemini's claim.

Item 6 — Channel robustness

The ITW gap is a domain problem, and the literature is consistent about the cause: existing corpora are dominated by clean read speech because TTS and VC training requires studio-quality audio [S5] — which describes LJ plus LibriSpeech plus VCTK exactly. The in-domain versus cross-domain AASIST spread on SpoofCeleb (0.61%/0.0160 validation vs 2.37%/0.0328 evaluation) [S5] is the mild version of this; the In-the-Wild study reports far larger degradations for benchmark-tuned systems [S18] (reported but not independently verified here).

The experts split on the lever. Gemini recommends RawBoost, which operates directly on raw waveforms to model telephony encoding, transmission and convolutive noise without requiring external data [S15] (reported but not independently verified). Claude recommends adding wild, non-read bona fide clips to the frozen-embedding path first. Two pieces of evidence favour Claude's ordering. First, the team's own post-band-fix numbers show that adding VCTK real speech did help on both axes once the 7.2 kHz confound was removed, so the diversity hypothesis is alive. Second, aggressive augmentation is documented to hurt SSL front-ends specifically: Fraunhofer SIT found extensive augmentation improved a non-SSL front-end but degraded an SSL one, and with wav2vec2-large, band-stop, parametric-EQ and tanh-distortion augmentation worsened minDCF in every configuration tested (0.066→0.081; 0.063→0.074; 0.097→0.144) [S16]. M1/M1b is a frozen XLS-R front-end from the same family. The same work found that adding a second corpus gave a marginal gain without augmentation (0.066→0.063) and a larger one with it (0.081→0.074), while piling on all available data was worse than the two-corpus setting [S16] — so add one good corpus, not three.

Concretely, the frozen path makes corpus addition nearly free: extract XLS-R layer-7 embeddings for ~4k new real clips (roughly 10 minutes at the measured 40-minutes-per-20k rate), speaker-group them into inner folds only, refit the logistic regression in seconds, and re-score holdout, ITW and test. Corpus choice should prioritise non-read, channel-diverse speech. SpoofCeleb is designed so that its bona fide speech is real-world noisy speech [S5][S14], with the authors' own caveat that the bona fide was enhanced and enhancement artifacts may confuse detector training [S5][S14]. VoxCeleb is the obvious alternative but its audio requires a request form and password [S20] (reported but not independently verified), a real same-day logistics risk — if nothing downloads cleanly within 30 minutes, abandon and fall back to augmentation rather than burning the morning.

Claude's untested hypothesis deserves a slot: a uniform ~7.2 kHz wall on every 16 kHz file is more characteristic of a low-bitrate codec round-trip than of naturally band-limited field audio, and a Kaiser low-pass matches the cutoff but not the quantisation noise, pre-echo, or psychoacoustic residue. If the test went through a codec and the training data did not, codec artifacts on real test speech can masquerade as vocoder artifacts — a clean mechanistic explanation for FA inflation. The test is cheap: encode 50 LJ clips at several bitrates, decode, compare rolloff slope and high-band spectral flatness against the test files, pick the closest match, apply the round-trip to all training clips, re-extract and refit. Apply it to both classes symmetrically — augmenting only the real class would teach "codec artifact implies real", the 7.2 kHz shortcut running in reverse. Keep the band-matching low-pass and avoid aggressive high-pass augmentation, since foundation-model front-ends are relatively robust to low-pass filtering but degrade sharply under high-pass [S6].

On the unlabelled test set: per-file z-norm and channel-statistic extraction are safe; the mixture-proportion $\hat\pi$ diagnostic is safe with the bias caveat; unsupervised CORAL or mean-shift on test embeddings is permitted but risky (a global affine match of a 30/70 mixture can shrink the class-discriminative direction) and should be validated train-to-ITW first and only shipped if it helps there; score normalisation to the test ECDF is safe but pointless because it is monotone. Both experts flag nearest-neighbour-to-LJ as a trap, and Claude gives the sharper reason: four of the ten DiffSSD generators are LJ-voice TTS that clone LJ's speaker, so LJ-proximity is a spoof-cluster indicator at least as much as a real-file indicator. Use it only to describe speaker composition in the README, never in the score path.

One-class and centre-loss objectives remain contraindicated. SAMO's authors report that bona fide speech from different speakers naturally forms multiple clusters in embedding space and that compacting them into one caused misclassification [S4] (reported but not independently verified). With one LJ speaker and ~80 LibriSpeech readers, the bona fide manifold is exactly the narrow multi-cluster case these objectives fail on.

Inference-time speech enhancement is documented to give measurable gains under noise and MP3 corruption [S6], but it adds a model to a CPU Docker image that is still unowned. Skip it today.

Item 7 — The non-speech gate

This is the best points-per-minute action available and both experts identify it, though only Claude sizes it. Silence currently scores 0.9996 and a synthetic music chord scores 1.0 — confidently wrong at the top of the ranking. On the test set, five such real files at the top add 0.040 DCF (~2.5 points), ten add 0.080 (~5 points), twenty add 0.160 (~10 points). The downside of gating a genuinely synthetic non-speech file to the bottom is 0.0020 DCF each — a 4:1 asymmetry in favour of gating. Against a fusion decision worth ~2 points, this is not close.

Procedure, about 30 minutes of CPU: run the existing VAD over all 1,671 test files and compute voiced fraction $v$ and total voiced duration $d$; plot the histogram (a measurement, not a labelling decision). Gate on $v<0.20$ or $d<0.5$ s or decode failure, routing the file into the pinned block at the minimum score. Add a cheap music guard — spectral flatness, harmonic-percussive ratio, and $f_0$ stability, with a sustained-chord signature (stable multi-$f_0$, near-zero formant movement) also routed to the block. Calibrate the threshold on training bona fide only, targeting under 0.5% trip rate on LJ and LibriSpeech, never on the test set. Log every gated file for the README. If the histogram shows zero files below threshold, the cost is 30 minutes and the return is a documented negative result that earns rubric credit.

Item 8 — Diversity and agentic orchestration

At 20% of the grade this is worth more than every remaining minDCF improvement combined, and the README is another 20%. Both experts agree the zero-signal engineered detectors should be documented honestly rather than fused. Keep them in the repo and in the orchestrator's routing trace but out of the score path: the rubric itemises container/metadata, ENF and splice explicitly, the README rubric rewards "what did not work", and a zero-AUC feature in a fused score adds variance for nothing. Document each null result with its mechanism — uniform FFmpeg decode destroys container evidence, 16 kHz mono PCM puts ENF's 50/60 Hz below the useful band after preprocessing, and re-encoding destroys splice discontinuities.

Replacements that survive a uniform decode, roughly in expected-signal order: noise-floor stationarity (real field audio has non-stationary background, TTS does not — and this doubles as a channel feature for $\hat\lambda$); speaker-embedding drift within a clip (cosine distance between first-half and second-half embeddings, a splice proxy); breath and pause statistics, since neural TTS under-generates breaths; microprosody ($f_0$ jitter, shimmer, contour high-frequency energy), since diffusion TTS over-smooths; phase and group-delay features, which the current magnitude-only handcrafted set omits entirely; and a Griffin-Lim round-trip distance, which is cheap and novel enough to anchor the explainability section.

For orchestration credit beyond static rules, a judge wants evidence that per-file routing changed the decision and that the delta was measured. Build a router that reads measured file properties (voiced fraction, duration, estimated SNR, effective bandwidth, speaker count) and selects the analysis chain: non-speech to the gate; short clips skip prosody and pause statistics for insufficient support; low-SNR or wild-channel files up-weight the SSL path and down-weight the magnitude-spectral detector; multi-speaker files trigger splice and drift analysis. Add an explicit abstention path into the default policy, emit a per-file JSON trace with each analysis's contribution, ship 5–10 worked examples, and report a router-on versus router-off ablation on holdout and ITW. Reporting a measured null delta is itself rubric-positive — the SZU-AFS team's published finding that fusion sometimes lands between the individuals with no improvement, so that using only the best classifier was sufficient [S17], is the model for that kind of honest negative result.

Item 9 — Ranked actions and lane allocation

Lane assignment through 11:00: the engineered-detector lane takes Docker, which is the highest-variance unowned deliverable at T−20h; the six idle CPU cores run the VAD histogram, the channel-statistic extraction, the holdout count decomposition and the codec-edge diagnostic; the local GPU runs the codec round-trip plus XLS-R re-extraction, then the wild bona fide extraction and the M3 perturbation probe; anyone with writing ability starts the README. The owner on waking ratifies the fusion rule from the A/B table, confirms the default policy, and owns the 14:00 draft.

On "the single change most likely to move holdout minDCF": the honest answer is none — holdout minDCF is already 0.015–0.044, a handful of files, and further movement there is noise rather than progress [S19]. The single change most likely to move the graded test minDCF is the non-speech gate plus codec-matched symmetric augmentation. The single change most likely to move the total score is resolving the score direction.

Failure modes not yet named by the team

The cost asymmetry inverts inside the sponsor's own script: the team's $9.333\times$ FA aversion is anti-aligned with the ASVspoof semantics of the shipped constants, where the $4\times$ weight falls on spoof-accepted. Add the miss-averse metric as a second reported column today. "Test share above 0.5" is a calibration statistic that cannot affect minDCF, and rules have been partly selected on it. The relevant decision threshold is 0.903 under the brief, not 0.5, so every intuition anchored on the half-way cut is off by roughly an order of magnitude in likelihood ratio. A handful of non-speech files can cost more than the entire fusion decision. Class-asymmetric augmentation would recreate the 7.2 kHz bug in reverse. There is no clean evaluation set for M3 anywhere in the team's possession, so holdout-only calibration does not contain the leakage risk. LJ-proximity is a spoof indicator as much as a real one. And Docker remains unowned with fewer than 20 hours left — a perfect TSV with no runnable image forfeits a large share of the grade.

What is measured versus what is not

Measured and solid: the DCF normalisation and its 1.00 ceiling; the per-file error costs on both holdout and test; the holdout's resolution floor; rankmean's dominance over zmean and stack_nonlj's dominance over M1b v3 on the two honest axes; the ITW $P_{miss}$ doubling under 50/50 fusion; handcrafted v4's 97% ITW $P_{miss}$; VCTK's positive post-band-fix effect on both axes; the non-speech scoring bug.

Unmeasured and not assumed either way: M3's provenance and therefore every M3 number including the pending 06:00 holdout; post-band-fix deep numbers for anything not yet re-run; the polarity and cost constants NSA will actually grade with; $\lambda$, the test's true domain mixture, which is the most decision-relevant unmeasured quantity and is 30 minutes away; whether the 7.2 kHz edge is a codec; whether a wild bona fide corpus can be downloaded and band-matched in time; and the magnitude of any wild-real-data gain in a frozen-embedding setup, which should not be expected to match figures from full in-domain training.

Scope Definition Q&A

• Q1:

  For the one-time draft review this Saturday afternoon, how many candidate TSVs can you actually submit to NSA, and is it strictly one shot (no resubmission if the first choice looks bad)?

  
  A:

  Exactly one draft TSV, one shot, no resubmission. NSA's instructions describe "one optional early draft submission for feedback." The feedback format is unknown; assume a single minDCF number, and assume it may be computed by their shipped ASVspoof5 code, so the returned value also tells us which score direction they use.

• Q2:

  Has the sponsor given any informal signal (even off-the-record) about which score direction (1.0=synthetic vs their shipped code's bona-fide-high convention) will actually be used for final grading?

  
  A:

  No signal, on or off the record. The question is posted in the NSA Discord channel and unanswered. At kickoff the sponsor said aloud that 1.0 means synthetic and that the test set is about 70% real; the shipped scoring code is the unmodified ASVspoof5 package with Pspoof = 0.5 and higher-score-is-bona-fide. We are submitting 1.0 = synthetic and will read the direction off the draft-review number: a value near 1.0 means they ran the code as shipped.

• Q3:

  Is there any budget or approval to acquire/use additional non-read-speech real audio (e.g., telephony, smartphone, or field-recording corpora) in the remaining hours, or must channel robustness be solved purely via augmentation of existing corpora?

  
  A:

  Budget and approval are not the constraint; the brief allows external datasets and we have network access. Time is the constraint: about 20 hours, and the fine-tune's data bundle is already frozen on the rented GPUs for its pilot and ablation, though a final run could take one more real source. On disk now: LJ Speech, LibriSpeech dev and test, ASVspoof 2019 LA bona fide (40 VCTK speakers, already used in M1b), MLAAD spoof, and In-the-Wild, which is evaluation-only by our own rule. A public wild-speech corpus of a few thousand real clips (VoxCeleb1 test or similar) could be downloaded and band-matched within an hour if you think real-world real speech in training beats augmentation. Every added corpus must be speaker-grouped into the inner folds only; the holdout stays fixed.

• Q4:

  Who has final authority to decide the shipped default-answer policy and fusion rule at the 14:00 cutoff — is it the single owner mentioned, or does the team need consensus/sign-off from others?

  
  A:

  One owner with full authority over fusion, the TSV and the draft submission; no consensus needed. He is asleep until about 11:00 EDT, so recommendations should be executable by the parallel lanes without him and presented as a decision he confirms on waking.

• Q5:

  What is the current best estimate of remaining CPU/wall-clock budget specifically for re-running M3 (Spectra-AASIST) holdout scoring and any additional augmentation experiments, beyond what's already queued?

  
  A:

  Local GPU (Apple M-series, MPS): occupied by the Spectra-AASIST full export until about 06:00, then free. Reference costs: scoring 21,671 clips with Spectra took 36 minutes; re-extracting XLS-R embeddings for the 20,000-clip training sample takes about 40 minutes. CPU: six cores idle from 06:00, handcrafted feature extraction for 20,000 clips takes 8 minutes on them. Cloud: $37 of vast.ai credit with three A100s running the fine-tune's pilot and ablation until a hard stop at 11:45, so roughly 15 spare A100-hours if a recommendation needs GPU training beyond that. Practically: several one-hour CPU experiments in parallel, GPU experiments serialized, one at a time.

• Q6:

  Is there anything else that would help refine this analysis?

  
  A:

  Fusion rules are cheap to add and evaluate: the fusion script already reads standardized logit columns and reports inner out-of-fold, holdout, per-real-source, In-the-Wild P_FA at the inner threshold and the test-set share above 0.5 in one run. Any rule you propose can be measured on all of those within minutes, so prefer recommendations we can A/B this morning over ones that need retraining. Spectra-AASIST's holdout numbers land around 06:00 and will be sent as a follow-up if you want them. The M5 fine-tune gate is 12:00; the draft TSV goes out at 14:00. propose as many rules as you see fit

Appendix A — Scope Definition Q&A

• Q1:

  For the one-time draft review this Saturday afternoon, how many candidate TSVs can you actually submit to NSA, and is it strictly one shot (no resubmission if the first choice looks bad)?

  
  A:

  Exactly one draft TSV, one shot, no resubmission. NSA's instructions describe "one optional early draft submission for feedback." The feedback format is unknown; assume a single minDCF number, and assume it may be computed by their shipped ASVspoof5 code, so the returned value also tells us which score direction they use.

• Q2:

  Has the sponsor given any informal signal (even off-the-record) about which score direction (1.0=synthetic vs their shipped code's bona-fide-high convention) will actually be used for final grading?

  
  A:

  No signal, on or off the record. The question is posted in the NSA Discord channel and unanswered. At kickoff the sponsor said aloud that 1.0 means synthetic and that the test set is about 70% real; the shipped scoring code is the unmodified ASVspoof5 package with Pspoof = 0.5 and higher-score-is-bona-fide. We are submitting 1.0 = synthetic and will read the direction off the draft-review number: a value near 1.0 means they ran the code as shipped.

• Q3:

  Is there any budget or approval to acquire/use additional non-read-speech real audio (e.g., telephony, smartphone, or field-recording corpora) in the remaining hours, or must channel robustness be solved purely via augmentation of existing corpora?

  
  A:

  Budget and approval are not the constraint; the brief allows external datasets and we have network access. Time is the constraint: about 20 hours, and the fine-tune's data bundle is already frozen on the rented GPUs for its pilot and ablation, though a final run could take one more real source. On disk now: LJ Speech, LibriSpeech dev and test, ASVspoof 2019 LA bona fide (40 VCTK speakers, already used in M1b), MLAAD spoof, and In-the-Wild, which is evaluation-only by our own rule. A public wild-speech corpus of a few thousand real clips (VoxCeleb1 test or similar) could be downloaded and band-matched within an hour if you think real-world real speech in training beats augmentation. Every added corpus must be speaker-grouped into the inner folds only; the holdout stays fixed.

• Q4:

  Who has final authority to decide the shipped default-answer policy and fusion rule at the 14:00 cutoff — is it the single owner mentioned, or does the team need consensus/sign-off from others?

  
  A:

  One owner with full authority over fusion, the TSV and the draft submission; no consensus needed. He is asleep until about 11:00 EDT, so recommendations should be executable by the parallel lanes without him and presented as a decision he confirms on waking.

• Q5:

  What is the current best estimate of remaining CPU/wall-clock budget specifically for re-running M3 (Spectra-AASIST) holdout scoring and any additional augmentation experiments, beyond what's already queued?

  
  A:

  Local GPU (Apple M-series, MPS): occupied by the Spectra-AASIST full export until about 06:00, then free. Reference costs: scoring 21,671 clips with Spectra took 36 minutes; re-extracting XLS-R embeddings for the 20,000-clip training sample takes about 40 minutes. CPU: six cores idle from 06:00, handcrafted feature extraction for 20,000 clips takes 8 minutes on them. Cloud: $37 of vast.ai credit with three A100s running the fine-tune's pilot and ablation until a hard stop at 11:45, so roughly 15 spare A100-hours if a recommendation needs GPU training beyond that. Practically: several one-hour CPU experiments in parallel, GPU experiments serialized, one at a time.

• Q6:

  Is there anything else that would help refine this analysis?

  
  A:

  Fusion rules are cheap to add and evaluate: the fusion script already reads standardized logit columns and reports inner out-of-fold, holdout, per-real-source, In-the-Wild P_FA at the inner threshold and the test-set share above 0.5 in one run. Any rule you propose can be measured on all of those within minutes, so prefer recommendations we can A/B this morning over ones that need retraining. Spectra-AASIST's holdout numbers land around 06:00 and will be sent as a follow-up if you want them. The M5 fine-tune gate is 12:00; the draft TSV goes out at 14:00. propose as many rules as you see fit

Appendix B — Project Metadata

Mode: Deep Think

Created: September 26, 2026

Expert models:

• Claude Opus 5 · Extra High
• Gemini 3.1 Pro · High

Generated by VeriLM