PROJECT
OVERVIEW
PROBLEM
A convincing synthetic voice can sound real. Calling real speech synthetic is the expensive error in this challenge.
SOLUTION
One decode path, ten forensic detectors, a fixed router, and an explained probability for every file.
MEASURED RESULTS
0.0065 holdout · 0.228 In-the-Wild normalized minDCF.
SCOPE
CORE SYSTEM
- Decode audio once to a shared format
- Run ten detectors behind one contract
- Fuse M1b, handcrafted, and M5 by rank
- Let Spectra only lower a likely false alarm
- Write a score and evidence report
OUT OF SCOPE
- No hardware signal or device attestation
- No hosted model on the scoring path
- No retraining during inference
Full documentation and sources
The numbers
All figures are normalized minDCFMinimum detection cost function. The scorer sweeps every possible threshold over our scores and reports the lowest cost it finds, with a false alarm weighted far more than a miss and the whole thing scaled so that 1.0 equals the best constant answer and 0.0 is perfect. Because the threshold is chosen by the scorer, only the order of our scores matters. (lower is better). The rule was chosen by a pre-declaredWritten down before any result was looked at: the candidates, the readouts, and the rule that picks a winner. Once the rule has been applied there is no second try. selection rule, applied once; the out-of-foldA score a model gives to a training row it never saw: the training data is split into folds, and each row is scored by the model fitted on the other folds. Every selection decision in this project uses out-of-fold scores. figure is the training-side readout, the holdoutA slice of the training data that no model was ever fitted, tuned or selected on: two generators and a set of real-speech groups. Each model reads it once. With only two held-out generators it is noisy, so small differences on it are not evidence. and In-the-WildA public set of web-sourced real and fake speech, used only as a stress test for false alarms on unfamiliar real voices. Nothing was ever trained on it. figures were each read once after the choice.
The cost function we are judged on
minDCF = 9.33 × PFA + 1 × Pmiss
One real file called synthetic costs as much as 9.33 missed fakes. That single ratio shaped every decision in this system.
- PFA
- false-alarm rate: the share of real files scored above the threshold
- Pmiss
- miss rate: the share of synthetic files scored below it
- min
- the best threshold is chosen by the scorer, so only the ranking of scores matters
Built from the sponsor's weights CFA = 4, Cmiss = 1 and a prior of 0.3 synthetic (about 0.7 of files are real): cost = 4 · PFA · 0.7 + 1 · Pmiss · 0.3 = 2.8 · PFA + 0.3 · Pmiss, divided by the best constant decision, 0.3, so that 1.0 means "no better than always saying real" and 0.0 is perfect. source: src/hearsay/metrics.py lines 29-31 (GitHub; assumes main is public)
How it decides, in one picture
One decode path, ten detectors behind one contract, a router that tells each result what it may do, and a fusion rule that reads only ranks: 0.6 × rank(M1b) + 0.2 × rank(handcrafted) + 0.2 × rank(M5), then Spectra-AASISTAn off-the-shelf anti-spoofing model run without any training. It is the strongest detector on every set we hold, but its training data is undisclosed, so it is allowed only to lower scores, never to raise them. allowed only to pull a file toward real, then a Platt mapA one-dimensional logistic fit that turns a raw decision value into a probability. It is monotone, so it changes the numbers a reader sees without changing the ranking the metric scores., with undecidable files in the pinned blockFiles the system cannot judge (no speech, or a failed decode) are placed strictly below every judged file, at the real end of the ranking, in a deterministic order. Under the sponsor's cost, an uncertain file belongs there. at the real end. The full walk-through, with a real test file, is on How it decides.
Mermaid source
%% The shipped pipeline, one file at a time. Numbers cited below.
%% cites: README.md#how-it-decides, README.md#orchestration-what-routing-changes, README.md#the-eight-forensic-techniques
flowchart TB
F["audio file · any container or codec"] --> L["probe the header, then decode once<br/>FFmpeg → 16 kHz mono float32"]
L --> C["ClipContext<br/>read-only audio + header facts + a per-clip cache"]
C --> R["router: fixed rules on measured facts<br/>every detector runs; each result is told what it may do"]
R --> FD["fused by rank<br/>XLS-R probe M1b · 0.6<br/>handcrafted spectral + prosody · 0.2<br/>M5 fine-tuned head · 0.2"]
R --> SD["suppression only<br/>Spectra-AASIST · can only halve a high score"]
R --> ED["evidence and routing, never scored<br/>compression · ENF hum · splice<br/>speaker drift · container"]
R --> G["speech gate<br/>is there speech to judge?"]
FD --> FU["base = 0.6 · rank(M1b) + 0.2 · rank(handcrafted) + 0.2 · rank(M5)"]
FU --> S["Spectra step: margin below −3 and base above 0.5 → halve"]
SD --> S
S --> PL["Platt map → probability<br/>judged files score in [0.001, 1]"]
G -->|"no speech, or the decode failed"| PB["pinned block · below 0.001<br/>at the real end of the ranking"]
PL --> T["CrossExam_predictions.tsv"]
PB --> T
ED --> X["explanation JSON per file<br/>every detector's sentence + the routing log"]
PL --> X
classDef fusedC fill:#00254b,stroke:#00254b,color:#ffffff
classDef suppC fill:#fff3d1,stroke:#c37530,color:#101a2b
classDef evC fill:#ffffff,stroke:#c37530,color:#101a2b,stroke-dasharray:4
classDef gateC fill:#e9eff6,stroke:#005493,color:#101a2b
class FD,FU fusedC
class SD,S suppC
class ED,X evC
class G,PB gateC
Results at a glance
| Inner OOF | Holdout | In-the-Wild, brief cost | In-the-Wild, sponsor-code cost | |
|---|---|---|---|---|
| Shipped rule (A3 w 0.2 + E: M1b, handcrafted and M5 by rank, Spectra suppression) | 0.135source: README.md, section results-at-a-glance | 0.0065source: README.md, section results-at-a-glance | 0.228source: README.md, section results-at-a-glance | 0.239source: README.md, section numbers |
| Previous rule (E on α 0.2source: README.md, section results-at-a-glance, no M5), the fallback | 0.140source: README.md, section results-at-a-glance | 0.014source: README.md, section results-at-a-glance | 0.260source: README.md, section results-at-a-glance | 0.267source: README.md, section numbers |
| M1bOur primary detector: a frozen XLS-R with a small logistic classifier on one hidden layer. M1b is the version whose training folds also include a set of extra real speakers. alone, the best single detector we trained | 0.301source: README.md, section results-at-a-glance | 0.072source: README.md, section results-at-a-glance | 0.343source: README.md, section results-at-a-glance | 0.296source: README.md, section numbers |
| Spectra-AASISTAn off-the-shelf anti-spoofing model run without any training. It is the strongest detector on every set we hold, but its training data is undisclosed, so it is allowed only to lower scores, never to raise them. off the shelf, a suppressorThe one-way role given to Spectra-AASIST: when it is confident a voice is real and the fused rank is high, the rank is halved. It can never raise a score, so if it is wrong the cost is a miss, never a false alarm. only | not fused: its training data is undisclosed, so no row of ours is provably out-of-sample for it | 0.065source: README.md, section numbers, possibly in-sample for the same reason | ||
The test set has 1,671source: README.md, section numbers files. The shipped rule scores 27.4%source: README.md, section numbers of them above 0.5source: README.md, section numbers against the roughly 30%source: README.md, section numbers synthetic the brief states (the orchestration ablation, a different readout, reports 27.5%source: README.md, section orchestration-what-routing-changes); the speech gateFive cheap cues that decide whether a file contains speech at all (not silent, some voiced frames, pitch that moves, loudness that moves, not spectrally flat). A file that fails goes to the pinned block instead of fusion. abstains on 0 of 1,671source: README.md, section the-eight-forensic-techniques. Turning the routerThe fixed rules that decide what each detector's result is allowed to do for a given file: fused into the score, allowed only to lower it, or recorded as evidence. Every decision is written to a per-file routing log. off moves the holdout from 0.0065source: README.md, section orchestration-what-routing-changes to 0.0200source: README.md, section orchestration-what-routing-changes and In-the-Wild from 0.229source: README.md, section orchestration-what-routing-changes to 0.274source: README.md, section orchestration-what-routing-changes.
Where to go next
How it decides
The pipeline, the router and the fusion arithmetic, on a real test file with its evidence sentences.
The metric and the default answer
What minDCF is, why a false alarm costs so much, why only ranking matters, and where the undecidable files go.
What worked and what did not
The honest log: every shortcut found, every idea rejected, with its number and its mechanism.
The ten detectors
One card each, mapped to the sponsor's rubric, with its role, its numbers and one real evidence sentence.
Validation
Folds by generator and speaker, the pre-declared sweeps, and the decoding table for the draft review.
Systems and subsystems
One spec page per part of the system, every parameter cited to the line that sets it.
The library
Every document in the repository, rendered: reports, run specs, audits, handoffs, consult records.
Reproduce it
The commands, verbatim, and where every artifact lives.