2026-09-26 channel robustness (plan review)

The library

Rendered from docs/reports/2026-09-26_channel-robustness-plan-review.md; the markdown file is the record.

Critical — A’s mean posterior is not generally a mixture proportion. D3 treats a balanced domain classifier’s mean prediction as λ̂. With overlapping domains, this depends on classifier separation and shrinks toward 0.5; it need not recover a true 0.3 mixture. Keep it as a descriptive score, use an appropriately adjusted estimator for λ̂, and test overlapping domains and endpoint mixtures—not only easily separated Gaussians.
Critical — A does not establish that its mixture model applies to test audio. The classifier learns from real speech only, but predicts on a test set containing roughly 30% synthetic speech; compression holes and spectral floors can distinguish synthesis as well as channel. Selecting the lowest-scoring 70% introduces detector-dependent selection rather than establishing real labels. Require class/source sensitivity checks and an “unidentifiable” outcome before using λ̂ against the 0.32 crossover. Novelty detection alone cannot resolve this confound.
Critical — D2’s claimed speaker isolation is false for LJ Speech. scripts/make_folds.py::group_key groups LJ by chapter, so reusing its group column spreads the same speaker across domain-classifier folds. Build separate speaker groups for this diagnostic without changing the canonical fold file, and report an LJ-excluded sensitivity analysis. Fit scaling inside each training fold and bootstrap reference speakers/groups rather than individual OOF rows; otherwise the advertised confidence interval understates dependence.
Critical — E compares M3 against an in-sample M1b/M5 baseline. D6 samples inner clips but scores them with shipped models. scripts/train_probe.py fits the final probe on all inner rows, so these are not held-out M1b predictions; M5 also trained on inner data. Use the corresponding fold-specific checkpoints for every clean/perturbed pair, or use an explicitly evaluation-only held-out cohort. Without this change, D9(a)’s relative degradation verdict has an uncontrolled training-exposure confound.
Critical — D7’s correlation drop cannot independently establish memorisation risk. Pooled M1b–M3 Spearman depends on class proportions, generator mix and score distributions, which differ between holdout and unlabeled test. Bootstrapping does not remove those differences. Remove this statistic as a standalone D9 trigger, or establish an appropriate composition-controlled reference; describe the raw gap as descriptive domain disagreement.
Critical — E’s readouts do not measure the claimed shipped suppression exposure. D8 calls M3 logit < −3 “exactly” the exposure, but pipeline.py and fuse_sweep_m5.py also require the fused base rank to exceed 0.5. Record the actual conjunction through the existing fusion implementation, or label the M3-only percentage as an upper bound. Add clean and perturbed minDCF/EER, fixed-inner-threshold false-alarm/miss rates, and both metric readings: AUC and spoof-only MLAAD misses cannot assess the cost-sensitive effect of keeping suppression.
Critical — Codec delay/padding can contaminate both B and E. compression.launder returns the decoded stream without alignment or length restoration; the existing compression test explicitly tolerates added samples. Those samples can alter spectral-floor statistics, trimming and Spectra’s padding behavior. Specify alignment/common-support handling and test MP3/AAC at both encoding rates, including short clips, float scaling and failed decodes. Otherwise measured “codec sensitivity” can be a duration/padding artifact.
Critical — Several acceptance calculations have undefined degenerate cases. D3 divides by the clean–wild median difference, D4 divides by test IQR, and Mahalanobis distance requires a covariance treatment that is unspecified. Constant or redundant features can therefore produce infinities or singular matrices. Define tolerances, excluded-column reporting, covariance regularization, and insufficient-data/undefined-verdict behavior; add tests so NaN comparisons cannot silently produce “kept.”
Critical — The plan omits the required submission/log disposition. Steps 10–14 produce reports and commits but neither a new TSV plus submissions/log.csv row nor an explicit blocker. Preserving the shipped TSV is correct, but does not satisfy the stated completion requirement. Add a timestamped artifact through the existing no-overwrite writer with its validation provenance, or explicitly report that this diagnostic-only rung produces no new submission and why.
Suggestion — “Resumable” caches need an identity and completeness contract. Define keys covering path, crop seed, perturbation/codec settings, feature version and model identity. Validate unique keys and expected coverage before computing verdicts; retain explicit failures and use paired complete observations for ΔAUC. Test interrupted writes and changed settings so a restart cannot silently combine incompatible runs.
Suggestion — Bound the incomplete and uncovered outcomes explicitly. The hard stop is stated, but the audit loop is unbounded and C’s approval wait precedes final reporting. Define a deadline for launching work, checkpoint/cancellation behavior, and an “inconclusive” verdict for unfinished E probes. Also record telephony and physical replay as untested; the current perturbation grid cannot support a general channel-robustness conclusion.