Systems and subsystems
The stress sets
Two public sets that nothing was selected on: In-the-Wild, web-sourced real and fake speech that no model ever trained on, for false alarms on unfamiliar voices; and MLAAD, synthetic speech from many generators, as training-only data for M5 and a miss-rate readout for the others.
In plain words
Our training real speech is all clean read speech, so a model can look excellent on the holdout and still call ordinary phone or web audio synthetic. In-the-Wild is the check for that, read for every model and never trained on. It is also where the equal-weight fusion fell apart and where the pre-declared rules did their selecting. MLAAD answers a different question: how often a detector misses a generator it has never seen.
What goes in and comes out
In: the sets' clips, prepared exactly like training rows (band match, trim, a seeded test-length crop). Out: minDCF under both cost readings and the false-alarm rate at each model's own inner-fold threshold; for MLAAD, the miss rate at that threshold.
Where it lives
Each detector's export carries its stress rows; the sweeps read them (fuse_sweep.py, fuse_sweep_m5.py); the M3 probes score MLAAD (m3_probes.py). The M5 data module refuses In-the-Wild paths in any training manifest.
Parameters and where each is set
- In-the-Wild: 3,000source: README.md, section validation clips, 2,000source: README.md, section validation real from 54 speakers and 1,000source: README.md, section validation fake; evaluation only (README). Crops use a fixed stress seed, and the M5 addendum was re-run "crop-fair" after its first run scored whole clips.
- MLAADA public multi-generator synthetic-speech set. Its English clips were training-only data for the M5 model and a miss-rate readout on unseen generators for the others.: 6,180source: README.md, section datasets English clips from 143source: README.md, section datasets models pulled during the event; for M5 a training-only spoof source capped at 12%source: CLAUDE.md, section ai-use-disclosure of the spoof side per batch and 120source: CLAUDE.md, section ai-use-disclosure clips per model, with the generator families that validate on a fold excluded from that fold's model; for the leakage probes, 572source: README.md, section what-did-not-work clips, four per model.
Status
In use since M1; both cost readings reported from the fusion sweep on.
Numbers
Every XLS-R probe sits at 0.34–0.41source: README.md, section what-did-not-work on In-the-Wild against 0.07–0.16source: README.md, section what-did-not-work on the holdout, and the handcrafted model reaches 1.00source: README.md, section what-did-not-work there. The shipped rule: 0.228source: README.md, section results-at-a-glance under the brief's cost, 0.239source: README.md, section numbers under the sponsor code's. On MLAAD, the miss rates: Spectra 14.9%source: README.md, section what-did-not-work, M1b 22.7%source: README.md, section what-did-not-work, M5 28.7%source: README.md, section what-did-not-work (it trained on MLAAD), handcrafted 56%source: README.md, section what-did-not-work.
Known limits
In-the-Wild stands in for "wild"; telephony and physical replay are not represented. Spectra-AASIST's authors evaluated on In-the-Wild, so its number there cannot be trusted. The λ̂ estimate says the test set is partly wild (0.51source: README.md, section what-did-not-work, with a wide interval), so neither validation set represents it alone.
Tests that pin it
tests/test_m5_manifest.py refuses In-the-Wild paths in a training manifest; tests/test_channel_robustness.py pins the MLAAD miss-rate readout's threshold search and the perturbation probes; tests/test_spectra.py pins the stress-seed crops.
Sources
README, "Validation"; README, "Datasets"; the channel-robustness report.