Systems and subsystems

The metric module

The normalized detection cost used for every selection and every reported number, plus an exact re-implementation of the sponsor's shipped scoring code, so each result can be read both ways.

In plain words

Everything on this site is measured with one function that mirrors the sponsor's rule: a false alarm costs 4×source: CLAUDE.md, section conventions a miss and about 70%source: CLAUDE.md, section conventions of files are real, scaled so that 1.0source: src/hearsay/metrics.py line 7 (GitHub; assumes main is public) is the best constant answer. The sponsor's actual script reads scores the other way round and moves the heavy weight to the missed fake, so a second function reproduces that script exactly. The long explanation is on the metric page.

What goes in and comes out

In: labels (1 = synthetic) and scores. Out: normalized minDCF at the best threshold, the cost at a given threshold, EER, the Bayes threshold on a log-likelihood ratio, and the sponsor-code reading with or without a flip.

Where it lives

src/hearsay/metrics.py.

Parameters and where each is set

• Constants: C_FA = 4, C_MISS = 1, PI_SYNTH = 0.3. The normalizer is the best constant decision, min(C_FA·(1−π), C_MISS·π) (metrics.py, lines 41–42), which makes the normalized cost 9.33source: src/hearsay/metrics.py line 10 (GitHub; assumes main is public)·P_FA + P_miss.
• min_cost sweeps every threshold from the ROC curve and caps the result at the constant decision, so it is never above 1 (metrics.py, lines 54–59). Only the ranking of scores can change it.
• bayes_llr_threshold = ln(C_FA/C_MISS) − logit(π): the log-likelihood ratio above which calling a file synthetic pays (metrics.py, lines 62–64); decision_logit shifts a calibrated LLR so that a fixed 50%source: src/hearsay/metrics.py line 69 (GitHub; assumes main is public) cut makes that decision, which is a posterior above 0.8source: src/hearsay/metrics.py line 69 (GitHub; assumes main is public) at the four-to-one cost.
• sponsor_min_dcf: the sponsor's constants (bona fide prior 0.5source: docs/reports/2026-09-26_fusion-sweep-predeclared.md, section readouts-for-every-candidate, the 4× on an accepted spoof) with a higher score read as bona fide, and a flip switch (metrics.py, lines 78–96). The sweeps call this reading the "miss-averse" cost.
• report returns EER, minDCF, the actual cost at the Bayes threshold, the same at an even prior, and both sponsor-code readings; by_group gives per-generator and per-source readouts.

Status

Built in the first hours; the sponsor-code twin was verified against the shipped script within 1e-9source: tests/test_audio_and_submission.py line 252 (GitHub; assumes main is public) for both polarities (a data-gated test).

Numbers

Every table on this site.

Known limits

The docstring records an open question about whether the kickoff slide's formula weights the false-alarm term by the attack prior; the brief's wording is what the constants implement, and the sponsor's script is reproduced separately.

Tests that pin it

tests/test_audio_and_submission.py: constant decisions normalize to one, a perfect scorer gives zero and the Bayes threshold at an even prior is ln 4, the sponsor-code twin matches the shipped script. tests/test_docs_consistency.py pins the constants stated in the documents to the module.

Sources

CLAUDE.md, "NSA metric and format"; the plan, "Scoring and deliverables"; Architecture §1.