The library

Rendered from docs/STATUS.md; the markdown file is the record.

HEARSAY: current state (Sat Sep 26, ~20:05; the file is final unless a ratified candidate replaces it)

A snapshot for teammates joining now. The plan, with its rules and timeline, is docs/plan.md. The architecture, with diagrams, is docs/architecture.md. The NSA brief is docs/nsa-challenge.md. Setup and repo rules are in CLAUDE.md.

What we're building

HEARSAY is software only; the hardware idea was dropped Friday night. It's an audio authentication system that takes any audio file and outputs the probability that it's synthetic (1.0 = synthetic), plus a plain-English reason.

The design:

• A rule-based orchestrator decides which forensic detectors to run on each file.
• Each detector returns a score and evidence.
• A logistic stacker fuses the scores.

How we're graded (NSA instructions, data/nsa/HEARSAY_HackGT2026_Instructions.pdf):

Deliverables (final DM before 8 AM Sunday):

• teamName_predictions.tsv: header filename<TAB>cm-score, all 1,671 rows in the template's order.
• A Docker image that runs inference offline. Not required (NSA to Nathan in person, Sat Sep 26 ~12:00: the submission is the TSV alone); ours is built and verified anyway (hearsay:20260926-0916) and stays in the README as reproducibility evidence.
• This repo's README.
• One optional early "draft" submission for feedback. Used: sent Sat 12:30 (CrossExam_predictions.tsv, the A3 w0.2 + E file, sha256 fb783076…); NSA returned minDCF 0.0733, EER 3.534% at ~15:30 (minDCF and EER only). Decoding: the 0.00–0.20 band, our direction (EER 3.5%, not 96.5%), ship unchanged. Field, per NSA at ~16:15: one team ahead at 0.0584 / EER 2.5%, a 0.95-point gap on the 60-point detection score under points = (1 − ours)/(1 − best) × 60. Record: docs/reports/2026-09-26_sponsor-questions.md, submissions/log.csv (15:40 row).

Where things stand

Data

Data is gitignored. Put it under data/ with the same paths as above (data/nsa/..., data/ljspeech, data/librispeech), or ask Nathan for a copy from his external drive.

Findings so far (the "what worked / what didn't" log starts here)

• The NSA scoring code runs "backwards" from their instructions. It's ASVspoof5 code that treats a higher score as real; the instructions say 1.0 = synthetic.
  • Our decision: follow the instructions (1.0 = synthetic) and never flip.
  • We log how our scores would come out under their code both ways. The draft submission will show which one they actually use.
• Shortcuts we found and removed:
  • Loudness. Test clips peak near full scale, while training real speech peaks around 0.5, so loudness alone predicted the label (AUC 0.66). Fixed by normalizing each input.
  • Leading silence. LibriSpeech real clips start with about 0.37 s of silence; test clips have about 0.06 s. Fixed with the same silence trim on train and test.
  • Clip length and tiling. Training clips are 5–9 s and test clips about 3.4 s. The old 4 s window repeat-padded short test clips, which put a splice seam only in test data. Fixed by embedding each clip at its real length, and training on crops drawn from the test length distribution.
• Container format is a trap. Some training fakes are MP3 or 22 kHz, while every test file is 16 kHz WAV. Any metadata or compression detector must be trained on clips converted to 16 kHz WAV first, or it will learn "MP3 = fake".
• The container is the label on the training data (measured): LibriSpeech = FLAC, LJ = 22 kHz WAV, DiffSSD = WAV or MP3, and PlayHT's MP3s carry the same FFmpeg encoder tag (Lavf58.29.100) as all 1,671 test files. So the container detector is never learned; it returns routing facts and a constant score here.
• The test set is low-passed at about 7.2 kHz and no training corpus is (not even the sponsor's resampled LJ clips). Everything above 7 kHz was a train-vs-test fingerprint; the handcrafted detector called 90% of the test set synthetic until hearsay.handcrafted.band_limit (a 71-tap Kaiser low-pass matching the roll-off) was applied to every clip, train and test. Now 42% of test files score above 0.5. The deep path got the same band match on Sat ~04:20 (commit 133e536: prepare_segment calls band_limit first), and M1's embeddings are being re-extracted; details and numbers in docs/reports/2026-09-26_cpu-detectors.md.
• The 7.2 kHz wall is not an MP3/AAC codec (Sat ~12:50, channel-robustness lane, action B). On 100 inner real clips, MP3 at 24–96 kbps and AAC at 32–64 kbps, at 16 and 44.1 kHz, alone or followed by our Kaiser low-pass, none reproduces the test set's edge: none satisfies the pre-declared match rule (distance to the test medians: our Kaiser low-pass 1.081; best codec, AAC 64 k + Kaiser, 1.045, 3% better where the pre-declared rule required 20%). MP3 at 16 kHz makes the wall far too steep (−66 dB vs the test's −44), 44.1 kHz encodes leave no wall, and no variant matches the test's near-zero local-hole rate. So the symmetric codec refit (C) does not run, by the rule set in advance, and no refit M1b column is coming; the wall looks like a resampling low-pass, which our band match already mirrors. outputs/channel/codec_match.json.
• λ̂, the test set's wild-domain weight: 0.51 (95% CI 0.12–0.64), corrected (Sat ~12:55). The first estimate (0.947) is withdrawn: its dominant feature, clip_floor_db, a whole-spectrogram 1st percentile, was reading the 7–8 kHz stopband that the method's own rule excludes, i.e. it measured the test files' double low-pass, not their channel. Without it: speaker-grouped domain classifier (holdout real vs In-the-Wild real) OOF AUC 0.84; λ̂ 0.51 overall, 0.58 on the 1,170 files the shipped rule calls real, 0.54 without LJ on the clean side; novelty share 1%. Controls (share read as wild): VCTK 24.7%, DiffSSD spoof 19%, inner real 10%, the same inner real after an MP3 round-trip 9%, so the "wild = codec-processed" reading is withdrawn too. Per feature the test set sits between the two domains on 11 of 21 statistics, beyond the wild end on noise-floor stationarity and flatness, and beyond the clean end on loudness spread and several floor and deep-hole statistics (lambda.json per_feature), so it is its own recording population rather than a blend. Duration-matched reference clips give 0.36, and rate-consistent variants 0.48–0.57. Reading: indeterminate, with an interval that includes the 0.32 crossover, so λ̂ cannot say which validation set represents the test set. For the shipped rule this does not matter: fusion_v2 beats fusion_v1 on both the holdout (0.0065 vs 0.014) and In-the-Wild (0.228 vs 0.260), so the choice is robust to λ. Run spec docs/specs/2026-09-26_channel-robustness.md; scripts/channel_lambda.py; outputs/channel/lambda.json.
• Our holdout is optimistic about real-world audio. On the eval-only In-the-Wild set, both M1 and M1b score minDCF 0.37–0.40 against 0.08–0.15 on the NSA holdout. Adding 40 VCTK read speakers (M1b) improved every holdout slice but raised In-the-Wild false alarms at the inner threshold from 1.2% to 5.7%. Channel robustness, not more read speech or more spoof data, is the lever; every deep detector now reports In-the-Wild false alarms as a gate.
• Leakage check is clean. No test file is an exact copy of anything in LJ Speech, the NSA LJ subset, or In-the-Wild.
• Public-data sanity check (not an NSA number): frozen XLS-R plus logistic regression gets minDCF 0.023 on ASVspoof 2019's unseen attacks. The same model scored pure silence as 99% synthetic, which is why the silence handling above matters.

How to help right now

1. Setup

git clone https://github.com/nrstough/hearsay && cd hearsay
uv sync
cp .env.example .env
uv run pytest -q          # expect all passing; tests needing data/weights skip cleanly

Requires uv and ffmpeg (brew install uv ffmpeg).

2. Build a detector against src/hearsay/detectors/base.py.

• Put it in src/hearsay/detectors/<your_detector>.py.
• run(ctx) returns a DetectorResult with:
  • score in [0, 1], higher = more synthetic
  • evidence: one plain-English sentence explaining the score
  • features: a few named numbers
• ctx.audio is 16 kHz mono float32 and read-only.
• Never use file dates or filenames. Git, zip and Docker rewrite them.

Proposed split (see CLAUDE.md):

3. Validate on the shared folds.

• Use outputs/manifests/nsa_train_sample.csv (paths and labels) and splits/nsa_folds.csv.
• Fit only on inner folds 0–4 and report the outer holdout. Never train on holdout rows.
• Score with hearsay.metrics.report(y, scores).
• Export scores as outputs/detector_scores/<name>.csv with columns path, fold, split (inner_oof | holdout | test), score, logit, so fusion can use them.

4. Deadlines (from docs/plan.md):

• Detector scores due Sat 19:00.
• Fusion freezes Sat 22:00.
• Final TSV Sun 05:00.
• DM sent before 08:00.

5. Working in the repo

• Small commits.
• git add only your own files.
• Keep uv run pytest -q green.
• Write down what didn't work too; it's graded.

Open questions (ask in the NSA Discord HEARSAY channel)

• Does the HexLabs environment have GPUs, and where are the data paths? That would decide whether we rent a cloud GPU for fine-tuning.
• Which prior and cost model will they actually score with? Their code uses Pspoof 0.5 with the 4× cost on accepted spoofs, which isn't the brief's "false alarm ×4 at 70% real".
• Team name for the TSV filename.