Systems and subsystems
The channel-robustness probes
Three diagnostic scripts run on Saturday afternoon: how wild is the test set, is the 7.2 kHzsource: README.md, section what-worked wall a codec, and is the off-the-shelf Spectra model memorising. Diagnostic only: no TSV, no rule change.
In plain words
The holdout and In-the-Wild disagree about every model, so which one the test set resembles matters. λ̂ estimates that. The wall in the test spectrum might have been a codec we could reproduce in training; the codec hunt checked. And Spectra-AASIST's training data is unknown, so three probes looked for the symptoms of memorisation instead of the cause. The results are on What did not work; this page is what the scripts do and how their rules were set before the runs.
What goes in and comes out
In: holdout real speech, In-the-Wild real speech, the test files, control sets, and for the probes the shipped models and constants. Out: JSON and CSV readouts under the outputs directory, and one verdict per pre-declared rule.
Where it lives
channel_lambda.py (λ̂), channel_codec.py (the codec grid), m3_probes.py (the Spectra probes); the pre-declared rules in the run spec.
Methods and where each is set
- λ̂. Label-free channel statistics (the compression features plus noise-floor level and stationarity, a decay proxy, flatness, loudness spread, pause fraction) on identically prepared clips; columns reading above 7 kHz dropped because the test files carry two low-passes. A logistic classifier separates holdout real speech from In-the-Wild real speech with speaker-grouped out-of-fold scoring; λ̂ is adjusted classify-and-count, refused when the classifier is too weak, with a cluster bootstrap by speaker (channel_lambda.py, lines 119–207). The crossover the consult named is 0.32source: scripts/channel_lambda.py line 34 (GitHub; assumes main is public). An identifiability rule (classifier AUC, control shares, an LJ-removed sensitivity) must pass before λ̂ is read.
- The codec grid. 100source: docs/reports/2026-09-26_channel-robustness.md, section b-is-the-72-khz-wall-a-codec inner real clips through MP3 and AAC at several bitrates and two encode rates, alone and followed by our own low-pass, compared with all test files on high-band statistics; a codec "matches" only if it is at least 20%source: docs/reports/2026-09-26_channel-robustness.md, section b-is-the-72-khz-wall-a-codec closer than the low-pass alone and closer on both hole statistics (channel_codec.py, lines 101–117).
- The Spectra probes. 500source: README.md, section what-did-not-work holdout clips under four label-preserving perturbations (an MP3 round-trip re-aligned by cross-correlation, white noise at 20 dB SNR, ±2%source: README.md, section what-did-not-work speed, a one-sample shift); 600source: docs/specs/2026-09-26_channel-robustness.md MLAAD clips planned for the unseen-generator miss rate (572source: README.md, section what-did-not-work run, four per model); and the Spearman agreement of the probe and Spectra on holdout versus test rows. The verdict rule: the suppression step is "at risk" if Spectra is markedly less stable than the probe, if the shipped rule damps more than 5%source: docs/reports/2026-09-26_channel-robustness.md, section e-is-m3-memorising-and-what-does-its-suppression-step-cost of unseen fakes, or if the step ever raises the fused holdout cost by more than 0.01source: docs/reports/2026-09-26_channel-robustness.md, section e-is-m3-memorising-and-what-does-its-suppression-step-cost (m3_probes.py, lines 125–142).
Status
Done at 13:20 on Saturday. The symmetric codec refit was pre-declared and not run because no codec matched; the wild real-speech corpus was dropped for the time box.
Numbers
λ̂ = 0.51source: README.md, section what-did-not-work (95%source: README.md, section what-did-not-work CI 0.12–0.64source: README.md, section what-did-not-work), after a first estimate of 0.947source: README.md, section what-did-not-work was withdrawn. The closest codec was 3%source: README.md, section what-did-not-work nearer than the low-pass. Spectra's mean |ΔAUC| 0.0008source: README.md, section what-did-not-work against 0.0041source: README.md, section what-did-not-work for the probe; MLAAD miss rate 14.9%source: README.md, section what-did-not-work. The new weak spot: at 20 dB white noise the handcrafted model's AUC falls from 0.998source: README.md, section what-did-not-work to 0.64source: README.md, section what-did-not-work and the shipped rule's holdout cost rises from 0.012source: README.md, section what-did-not-work to 0.269source: README.md, section what-did-not-work.
Known limits
λ̂ rests on a handful of hole statistics and is a weak, single-method estimate; the identifiability rule passed only just. The agreement gap between the probe and Spectra is descriptive, not evidence of memorisation, because the two sets differ in class mix and generators.
Tests that pin it
tests/test_channel_robustness.py, hermetic: statistics finite on degenerate inputs and level-invariant, noise raises the floor, a reverb tail slows the decay, grouped folds never split a group and the leak check can fail, the adjusted estimate recovers known mixtures and refuses a weak classifier, the mean posterior's bias, the identifiability rule fails on each guard alone, novelty share, each perturbation changes the signal and none does not, noise hits 20 dB, the shift is one sample, speed changes length by two percent, the codec alignment restores length, ΔAUC sign, the verdict rule on each guard alone, the high-band statistics see the wall, the match rule needs distance and both hole statistics.