HEARSAY audio authentication, documented
GitHub

The story

Validation: how we know what we know

Every number on this site comes from a split that holds out whole generators and whole speakers, never single clips. Selection uses only out-of-fold scores; the holdout is read once per model; In-the-Wild is never trained on; the test set is unlabeled and only its share above 0.5source: README.md, section validation is watched. The fusion rule was chosen by rules written down before the results existed.

Folds by generator and speaker, never by clip

The fold scheme: 20,000 training rows split into five inner folds that each hold out whole generators, one outer holdout read once, stress sets never trained on, and the unlabeled test set.
The fold scheme: 20,000 training rows split into five inner folds that each hold out whole generators, one outer holdout read once, stress sets never trained on, and the unlabeled test set. Open full size
Mermaid source
%% How the data is split: by generator and by speaker, never by clip.
%% cites: docs/architecture.md#8-validation-and-selection, README.md#validation
flowchart TB
  FF["fold file · 20,000 training rows<br/>fakes grouped by generator · real speech by speaker"]
  subgraph inner["five inner folds · 16,142 rows · each holds out whole generators"]
    f0["fold 0 · grad_tts, unit_speech"]
    f1["fold 1 · diffgan_tts, openvoicev2"]
    f2["fold 2 · pro_diff, xtts_v2"]
    f3["fold 3 · your_tts"]
    f4["fold 4 · elevenlabs"]
  end
  HO["outer holdout · 3,858 rows<br/>playht + wavegrad2 + 26 real-speech groups<br/>read once per model, never selected on"]
  ST["stress sets · never trained on<br/>In-the-Wild: 2,000 real, 1,000 fake"]
  TE["NSA test set · 1,671 files · unlabeled<br/>only the share above 0.5 is watched"]
  FF --> inner
  FF --> HO
  inner -->|"every learned piece fit fold-locally:<br/>scalers, Platt maps, fusion weights"| OOF["out-of-fold scores<br/>the only selection signal"]
  HO --> HC["holdout readout"]
  ST --> SR["false-alarm readout on<br/>unfamiliar real speech"]
  TE --> TC["the TSV"]
  classDef data fill:#e9eff6,stroke:#005493,color:#101a2b
  classDef never fill:#fff3d1,stroke:#c37530,color:#101a2b
  class FF,f0,f1,f2,f3,f4,OOF data
  class HO,ST,TE never
  • The fold file has 20,000source: README.md, section validation rows: 10 DiffSSD generators, LJ Speech and LibriSpeech. Fakes are grouped by generatorA text-to-speech or voice-cloning system that produced synthetic clips. Folds hold out whole generators so that every readout is on synthetic speech from a system the model never saw.; real speech by speaker (LJ, a single speaker, by chapter). Five inner foldsThe five training folds. Each holds out whole generators, so a model is always tested on synthetic speech from a system it never trained on. Scalers, calibration and fusion weights are fitted inside each fold as well. each hold out whole generators: grad_tts + unit_speech, diffgan_tts + openvoicev2, pro_diff + xtts_v2, your_tts, ElevenLabs. Every learned piece of a detector, scalers and Platt maps included, is fit fold-locally, and selection uses only inner out-of-foldA score a model gives to a training row it never saw: the training data is split into folds, and each row is scored by the model fitted on the other folds. Every selection decision in this project uses out-of-fold scores. scores.
  • One outer holdoutA slice of the training data that no model was ever fitted, tuned or selected on: two generators and a set of real-speech groups. Each model reads it once. With only two held-out generators it is noisy, so small differences on it are not evidence., read once per model: PlayHT and WaveGrad2 plus 26 real-speech groups (3,858source: README.md, section validation rows). Nothing was ever trained, tuned or selected on it. Extra training data (ASVspoof 2019 real speakers, MLAAD, the rest of DiffSSD) enters inner folds only, so holdout numbers stay comparable across every model.
  • The holdout has a resolution floor. With two held-out generators its noise is about ±0.07–0.10source: README.md, section validation, so we treat any gap under 0.15source: README.md, section validation as noise. It can tell a good detector from a bad one, not a good fusion rule from a slightly better one.
  • In-the-WildA public set of web-sourced real and fake speech, used only as a stress test for false alarms on unfamiliar real voices. Nothing was ever trained on it. is the stress test: 3,000source: README.md, section validation clips (2,000source: README.md, section validation real from 54 speakers, 1,000source: README.md, section validation fake), evaluation only, never trained on. For each model we report minDCF and the false-alarm rate at the threshold chosen on inner folds, under both cost weightings.
  • The test set as a smoke alarm. We track the share of test files scored above 0.5source: README.md, section validation against the ~30%source: README.md, section validation the brief states. It flagged the band mismatch. It is never a selection signal: any monotone rescaling changes it without changing minDCF.

Source: the README's validation section.

The pre-declared sweeps

The fusion rule got where it is in two steps, each written down before any result was run. Source: "The fusion rule" in the README and the sweep report.

Step 1, frozen at 08:13

Step 2, the M5 addendum, ratified at noon

The day, in order

Times are Saturday, September 26, local to the event. Each entry links to the document that records it; where documents disagree by a few minutes the README's time is shown.

  1. First fusion: the two detectors averaged equally. Best holdout we had; doubled the In-the-Wild misses. README
  2. The rule frozen by the pre-declared sweep: α = 0.2source: README.md, section the-fusion-rule with Spectra suppression. README
  3. Two M5 candidates and a stricter replacement rule written down, before any run. README
  4. Addendum results appended; A3 w 0.2 + E qualifies; F does not. sweep report
  5. Held: the previous rule stays the draft-review payload; the runner side is built behind a flag. README
  6. Nathan ratifies the switch to A3 w 0.2 + E. README
  7. The draft sent to NSA by Discord DM as CrossExam_predictions.tsv. README
  8. The runner's default switched to the shipped constants file. STATUS
  9. The channel-robustness lane done: λ̂ corrected, the codec hunt negative, Spectra's suppression kept, the noise weakness found. STATUS

The draft review

For the draft reviewThe sponsor's one-time offer to score an early submission and return its minDCF. We sent the exact file we ship and decided in advance what each returned value would mean., we sent the exact TSV we ship on Saturday at 12:30 and decided in advance what each returned number would mean. NSA returned minDCF 0.0733source: README.md, section validation and EER 3.534%source: README.md, section validation. The minDCF fell in our pre-declared 0.00–0.20source: README.md, section validation band, so the rule stayed unchanged; the decoding table is on the metric page. Source: the README.