The story

What worked and what did not

The honest log. Each card names what we did, what it did, and the mechanism behind the number. The negatives are here on purpose: five shortcuts in the data, an equal-weight fusion our holdout loved and we rejected, a fine-tune that failed its gate, four detectors that cannot work on this test set, a codec hunt that came back empty, and an estimate we published and then withdrew.

What worked

• Band-matching the training audio to the test set

  The test set is low-passed at about 7.2 kHzsource: README.md, section what-worked and no training corpus is. A 71-tap Kaiser low-pass at 7.25 kHzsource: README.md, section what-worked on every clip, train and test, took the handcrafted detector from calling 90%source: README.md, section what-worked of test files synthetic to 42%source: README.md, section what-worked. Without it, everything above 7 kHz was a train-versus-test fingerprint.

• Real-speaker diversity, once band-matched

  Adding 40 VCTK speakers (M1bOur primary detector: a frozen XLS-R with a small logistic classifier on one hidden layer. M1b is the version whose training folds also include a set of extra real speakers.) improved the holdout from 0.159source: README.md, section what-worked to 0.072source: README.md, section what-worked and In-the-Wild from 0.380source: README.md, section what-worked to 0.342source: README.md, section what-worked. Before band matching, the same addition made In-the-Wild worse (0.374source: README.md, section what-worked → 0.405source: README.md, section what-worked): the wall was hiding the gain.

• Length parity

  Test clips are about 3.4 ssource: README.md, section what-worked and training clips 5–9 s. Embedding each clip at its own length and training on crops drawn from the test-length distribution removed a repeat-padding seam that only test clips had; it was worth 0.05source: README.md, section what-worked on its own for the handcrafted detector (0.709source: README.md, section what-worked → 0.661source: README.md, section what-worked).

• Gradient-boosted trees over logistic regression

  For the handcrafted features: holdout 0.374source: README.md, section what-worked vs 0.661source: README.md, section what-worked on the same features. The treesAn ensemble of small decision trees trained one after another. LightGBM trains them; at inference the dumped trees are evaluated in numpy so that the training library is never imported. transfer to unseen generators and the linear model does not.

• New feature families that see new generators

  CQCC frame-to-frame change and LFCC upper-coefficient variability made pro_diff visible (1.00source: README.md, section what-worked → 0.32source: README.md, section what-worked out of fold) and cut inner minDCF from 0.614source: README.md, section what-worked to 0.434source: README.md, section what-worked.

• Augmented twins for the handcrafted model

  Codec round-trips and random spectral tilt, added as fit-only extra rows, cut holdout-real false alarms from 9.1%source: README.md, section what-worked to 6.3%source: README.md, section what-worked and In-the-Wild false alarms from 0.65%source: README.md, section what-worked to 0.40%source: README.md, section what-worked, and improved the holdout (0.170source: README.md, section what-worked → 0.137source: README.md, section what-worked). The price was inner minDCF (0.418source: README.md, section what-worked → 0.469source: README.md, section what-worked) and pro_diff (0.30source: README.md, section what-worked → 0.63source: README.md, section what-worked). With a false alarm worth 9.33source: README.md, section what-worked misses, we took the trade. The model is trained with augmented twins and validated on clean rows only.

• Treating the container as routing, not evidence

  We measured it before using it and found it was the label (below), so the detector never learns from it.

• Laundering both classes for compression forensics

  Re-encoding a random half of both classes through MP3/AAC meant the detector could not learn "MP3 = ElevenLabs/PlayHT". It then detects codec history well (AUC 0.90source: README.md, section what-worked–0.94source: README.md, section what-worked).

• A pre-declared fusion sweep

  Writing the candidates and the selection rule down first stopped us from shipping the equal-weight rule our holdout liked best.

• M5 as a minority vote

  Alone it lost to the plain probe on every holdout readout. Given 20%source: README.md, section what-worked of the rank blend beside M1b and the handcrafted model, it cut the fused holdout from 0.014source: README.md, section what-worked to 0.0065source: README.md, section what-worked and In-the-Wild from 0.260source: README.md, section what-worked to 0.228source: README.md, section what-worked. It errs on different files than M1b, and that is what fusion uses.

• Spectra-AASIST as a one-way vote

  As a false-alarm suppressor it moved In-the-Wild from 0.322source: README.md, section what-worked to 0.260source: README.md, section what-worked at no cost elsewhere. Used that way, if it is wrong about a file the cost is a miss, never a false alarm. The later leakage probes found the step fires only on real clips (4–5%source: README.md, section what-worked of them) and never on a fake, under every perturbation we tried.

• The phase cues as evidence

  Peak phase-versus-magnitude coherence separates grad_tts from real speech (AUC 0.80source: README.md, section what-worked), is corpus-neutral, and names a vocoder property a person can check.

What did not work

• Five shortcuts in the data, all found and closed

  Level: test clips peak near full scale and training real speech near 0.5source: README.md, section what-did-not-work; peak alone predicted the label (AUC 0.66source: README.md, section what-did-not-work). Closed by normalizing each input. Leading silence: LibriSpeech starts with about 0.37 ssource: README.md, section what-did-not-work of silence, test clips 0.06 ssource: README.md, section what-did-not-work. Closed by trimming both. Tiling: repeat-padding short test clips put a seam only in test data. Closed by length parity. Container: every test file is 16 kHz PCM WAV with the same FFmpeg tag. Closed by one decode path. The 7.2 kHzsource: README.md, section what-did-not-work wall: closed by band matching.

• The codec hunt came back empty

  We checked whether the wall is a codec, to reproduce it more faithfully in training. It isn't: across 34 MP3 and AAC settings none matched it better than our own Kaiser low-pass (the closest was 3%source: README.md, section what-did-not-work nearer on distance and worse on spectral holes). It is a smooth low-pass, like a resampler's anti-alias filter, so the planned symmetric codec refit never ran.

• The container is the label

  LibriSpeech is FLAC, LJ is 22 kHz WAV, and DiffSSD's ElevenLabs and PlayHT are MP3. PlayHT's MP3s carry the exact encoder tag that all 1,671source: README.md, section what-did-not-work test files carry. A learned metadata model would score perfectly on our holdout for the wrong reason, then call the whole test set PlayHT. The container detector returns a constant 0.5source: README.md, section what-did-not-work on this test set by design.

• Compression forensics sees the pipeline, not the class

  With codec history equalized, class holdout minDCF is 0.90source: README.md, section what-did-not-work, and it calls 81%source: README.md, section what-did-not-work of the test set synthetic, because the test set reads as laundered throughout (mean P(laundered) 0.62source: README.md, section what-did-not-work). It stays as evidence.

• ENF cannot be validated here

  Stable mains hum is in 36%source: README.md, section what-did-not-work of LJ Speech clips (a home studio) and 21%source: README.md, section what-did-not-work of LibriSpeech, and in almost no generated audio, so a learned version would learn "hum = LJ = real". Only 24 test files carry hum. Rule-based and mild on purpose; evidence only.

• Splice finds vocoder artifacts, not edits

  On training data the seams are single-sample clicks in 27%source: README.md, section what-did-not-work of WaveGrad2 and DC jumps in 9–13%source: README.md, section what-did-not-work of several generators and 9%source: README.md, section what-did-not-work of LibriSpeech. The sponsor said no test clip is partially synthetic. It flags 58 test files; evidence only.

• Speaker drift points the wrong way

  Fakes are the most self-consistent voices: mean window-to-window similarity separates them toward fake at AUC 0.73source: README.md, section what-did-not-work. A cloned voice doesn't vary from second to second the way a person does. Drift flags 16%source: README.md, section what-did-not-work of test files, mostly the hard real recordings. Fused, it would push those toward synthetic, so it is evidence only.

• Fine-tuning XLS-R lost to a frozen probe

  Fine-tuning all 12 kept layers fit the seen generators and transferred worse to the unseen commercial one: 0.38–0.41source: README.md, section what-did-not-work on the ElevenLabs fold vs 0.31source: README.md, section what-did-not-work with the backbone frozen. The frozen-backbone head then lost to the plain probe on the holdout (0.363source: README.md, section what-did-not-work vs 0.159source: README.md, section what-did-not-work), on WaveGrad2, on LibriSpeech real speech and on clips under 4 s (0.60source: README.md, section what-did-not-work). It won only on In-the-Wild false alarms (0.45%source: README.md, section what-did-not-work vs 0.7%source: README.md, section what-did-not-work). Its one use is as a 20%source: README.md, section what-did-not-work voice in fusion. Twelve A100 instances, $5.42source: README.md, section what-did-not-work in total.

• Pruning corpus-cue columns hurt every time

  83 handcrafted columns separate LJ from LibriSpeech more than they separate real from fake. Dropping them made every readout worse, including LibriSpeech false alarms (0.54source: README.md, section what-did-not-work → 0.59–0.61source: README.md, section what-did-not-work); stripping v3's own cue columns took it from 0.614source: README.md, section what-did-not-work to 0.848source: README.md, section what-did-not-work. The check told us which per-source numbers to read, not which columns to delete.

• The handcrafted detector's test-domain offset

  Even band-matched, the test recordings have a darker, codec-like spectral envelope. The 234source: README.md, section what-did-not-work columns tell train from test at AUC 0.99source: README.md, section what-did-not-work, and 84%source: README.md, section what-did-not-work of the shifted columns move toward "fake". Pruning the shifted columns made it worse (the rest still separate at 0.98source: README.md, section what-did-not-work); augmentation removed only part of it. This is part of why it gets 20%source: README.md, section what-did-not-work of the vote, not half.

• grad_tts is invisible to the handcrafted model under our validation

  Its phase cue is shown by no other generator, so when grad_tts is the held-out one, nothing teaches the cue: 1.00source: README.md, section what-did-not-work in every variant. That is an honest limit of holding out by generator.

• Prosody beyond pitch

  Pause fraction and count, voiced fraction, loudness dynamics and zero-crossing rate all sat within 0.05source: README.md, section what-did-not-work of chance.

• Equal-weight fusion: rejected

  Our first fusion averaged the two detectors equally. On the holdout it looked like the best thing we had: 0.018source: README.md, section the-fusion-rule and 0.015source: README.md, section the-fusion-rule, against 0.072source: README.md, section the-fusion-rule for M1b alone. On In-the-Wild it doubled the misses, from 28%source: README.md, section the-fusion-rule for M1b alone to 58–60%source: README.md, section the-fusion-rule, and pushed the share of test files called synthetic up to 32–40%source: README.md, section the-fusion-rule. The holdout gain was below the holdout's resolution floor; the In-the-Wild loss was large and consistent. The handcrafted column is excellent on generators it has seen and blind to web-sourced fakes (97%source: README.md, section the-fusion-rule missed on its own), so giving it half the vote traded real-world detection for a holdout number.

• Spectra kept out of the vote, though a full vote scores better

  With M5 in the blend, fusing everything equally beats the shipped rule on every labeled readout (holdout 0.0045source: README.md, section orchestration-what-routing-changes, In-the-Wild 0.214source: README.md, section orchestration-what-routing-changes and 0.196source: README.md, section orchestration-what-routing-changes). It does so by giving Spectra-AASIST a full vote. We ruled that out before seeing these numbers, because Spectra's training data is undisclosed and our validation rows may be in its training set. The choice rests on that argument, not on these readouts.

• Our holdout overstates real-world skill

  Every XLS-R probe we trained sits at 0.34–0.41source: README.md, section what-did-not-work on In-the-Wild against 0.07–0.16source: README.md, section what-did-not-work on the holdout, and the handcrafted model gets 1.00source: README.md, section what-did-not-work there. Channel robustness, not more read speech or more fake data, is the gap we did not close. A classifier separating our holdout's real speech from In-the-Wild's, run over the test files, puts the wild share at λ̂ = 0.51source: README.md, section what-did-not-work (95%source: README.md, section what-did-not-work CI 0.12–0.64source: README.md, section what-did-not-work). The test set is partly wild, so neither validation set represents it alone; the shipped rule beats the previous one on both, so the choice does not depend on λ.

• A pretrained model we can't audit, then probed

  Spectra-AASIST separates our inner rows perfectly (AUC 1.000source: README.md, section what-did-not-work). Its model card names no training set, so its inner, holdout and In-the-Wild numbers may all be in-sample. We probed for the symptoms of memorisation instead: on 500source: README.md, section what-did-not-work holdout clips under four label-preserving changes it was the most stable detector (mean |ΔAUC| 0.0008source: README.md, section what-did-not-work, against 0.0041source: README.md, section what-did-not-work for M1b); on 572source: README.md, section what-did-not-work MLAAD clips from 143source: README.md, section what-did-not-work generators it had the lowest miss rate (14.9%source: README.md, section what-did-not-work, against 22.7%source: README.md, section what-did-not-work for M1b, 28.7%source: README.md, section what-did-not-work for M5, 56%source: README.md, section what-did-not-work for the handcrafted model); its suppression step never raised the fused holdout cost. Verdict: keep it as a suppressor only. The one number we still can't trust is its In-the-Wild score, because its authors evaluated on that set.

• Peak-normalizing Spectra's input

  A fix for its within-fake level correlation cost 0.033source: README.md, section what-did-not-work on real speech and did not remove the correlation, so the level was a proxy, not the cue. Rejected.

• Additive noise is the shipped rule's weak spot

  At 20 dB SNR white noise on 500source: README.md, section what-did-not-work holdout clips the handcrafted model loses almost all discrimination (AUC 0.998source: README.md, section what-did-not-work → 0.64source: README.md, section what-did-not-work, minDCF 0.048source: README.md, section what-did-not-work → 0.996source: README.md, section what-did-not-work); M1b rises from 0.056source: README.md, section what-did-not-work to 0.271source: README.md, section what-did-not-work, and the shipped rule from 0.012source: README.md, section what-did-not-work to 0.269source: README.md, section what-did-not-work. Spectra holds at 0.116source: README.md, section what-did-not-work, but noise pushes it out of its "strongly real" region, so its suppression step never fires. Found after the rule was frozen and not fixed; the pointer is noise augmentation of the handcrafted training, applied to both classes.

• A wild real-speech corpus, planned and not attempted

  It was the fusion consult's suggested fix for the In-the-Wild gap. We dropped it at 12:16 on Saturday because a download, extraction and refit would not fit the time box, and the time went to the Spectra probes and the λ estimate. The two candidates looked unusable anyway: VoxCeleb1's test set needs a request form, and SpoofCeleb's real speech has been speech-enhanced.

• Our first estimate of how wild the test set is was wrong

  It said λ̂ = 0.947source: README.md, section what-did-not-work with a classifier AUC of 0.99source: README.md, section what-did-not-work, and read "wild" as "codec-processed". One feature did the work: a whole-spectrogram noise-floor percentile that landed in the 7–8 kHz stopband. It was measuring the test files' double low-pass, not the recording channel: one MP3 round-trip moved 91%source: README.md, section what-did-not-work of our clean clips to "wild". Dropping that feature, under the rule we had already declared for columns above 7 kHz, gave 0.51source: README.md, section what-did-not-work (MP3 control 91%source: README.md, section what-did-not-work → 9%source: README.md, section what-did-not-work). The same kind of shortcut as the 7.2 kHzsource: README.md, section what-did-not-work wall, read through a feature that didn't look like a band feature.