2026-09-26 m5 xlsr finetune (plan review)

The library

Rendered from docs/reports/2026-09-26_m5-xlsr-finetune-plan-review.md; the markdown file is the record.

Critical — Training preprocessing uses validation data. LengthMixer.cap is computed from the entire labeled bundle, and bundle-wide shortcut diagnostics determine whether training proceeds. That includes the outer holdout and each model’s validation fold. Compute caps, matching statistics and training gates exclusively from training_rows(m, fold); test that changing held-out labels or durations cannot change training configuration.
Critical — Cross-dataset leakage checks are missing. Path deduplication and generator-name exclusions do not detect identical audio under different paths or public-data speaker overlap. docs/plan.md explicitly requires audio-hash deduplication, and scripts/hash_audio.py already exists. Audit extras against core validation/holdout and test audio before training, enforce known speaker exclusions, and document unresolved voice overlap.
Critical — The cloud bundle is not self-contained. test_duration_sampler() reads splits/nsa_test_durations.csv, which the bundle omits; fold-file verification also needs splits/nsa_folds.csv. Setup does not explicitly install the packaged hearsay source. Include the required split artifacts and install/import configuration, then run a smoke test using only the extracted bundle outside the repository.
Critical — The shortcut gate misses strong shortcuts and cannot handle MLAAD. AUC below 0.15 is just as predictive as AUC above 0.85, while MLAAD contains only spoof, making within-source label AUC undefined. Bundle-level measurements also miss changes caused by cropping and augmentation. Use max(AUC, 1−AUC), explicitly handle single-class strata, measure the actual training stream, and implement the spec’s omitted MLAAD-versus-LibriSpeech corpus probe.
Critical — Allowed dropped rows contradict mandatory output counts. The bundle permits short clips and up to 0.5% decode failures to disappear, but assembly requires every core and test row. Restrict dropping to training extras; preserve required row identities and define flagged fallback scoring or an early blocking failure. Test corrupt and short core/test files through assembly.
Critical — Fusion scores use different inference policies. Inner OOF scores come from fixed short crops, while test scores use whole clips up to 14 seconds; assembly does not specify which holdout stream enters fusion. Export OOF, holdout and test scores using the same deployment transform. Keep cropped and augmented evaluations as separate diagnostics.
Critical — The pilot cannot reliably select E or exercise resume. A 300-step pilot may finish before its first epoch checkpoint, and validation every 250 steps provides only one scheduled readout. Nevertheless, the plan requires an R2 resume and selects an epoch count from this run. Define epoch length, checkpoint at step limits and normal exit, and choose a consistent step/epoch budget with enough validation checkpoints to support selection.
Critical — The proposed disk allocation is insufficiently sized. The plan estimates 6.7 GB per audio variant; clean audio plus five variants already approaches 40 GB before weights, optimizer states, downloads and temporary codec files. Size storage from a measured pilot with headroom, or generate a bounded subset of variants and remove temporary files promptly.
Critical — Bundle identity and resumption can accept stale artifacts. Existing FLACs are skipped without validating their provenance; the tree hash covers neither manifests nor code/configuration; checking 200 files cannot verify the whole tree. Reducing the manifest also will not reduce a recursive rclone copy of existing files. Use immutable bundle versions, manifest-driven transfer lists, verified resumable outputs, and hashes covering all training inputs.
Critical — Cloud cleanup does not cover launcher failures. The destruction trap is installed after pip install, and failures between instance creation and starting box_chain.sh leave no protected cleanup path. A box-local watchdog also cannot handle an unreachable machine. Add launcher-side cleanup, an external deadline/reaper, and aggregate budget accounting across concurrent instances; verify these with mocked failure-path tests.
Critical — The reduced fallback violates the output contract. Fold-4-only scoring cannot satisfy the required 16,142 OOF rows. Fold 4 also selects the checkpoint, so it is not an untouched stacking set. Specify a separately agreed fusion interface and independent selection split for this fallback, or stop and report M5 blocked without emitting a nominally complete score file.
Critical — Submission completion has no concrete handoff. The spec assigns TSV creation to the main chat, but this plan ends without a verified new TSV/log row or a named blocking outcome. Add an explicit owner and completion check using write_submission and append_log, including preflight and template-order validation, while preserving previous submissions.
Critical — Secret handling conflicts with project policy. Reading the Vast key from ~/.config/vastai/vast_api_key and sending inline rclone configuration does not satisfy the stated .env-only rule. Specify an approved credential-injection mechanism, prevent credentials entering bundles/logs, and define remote cleanup.
Suggestion — Verify raw-audio deployment parity, not only device parity. Comparing CPU and GPU on the same FLAC bundle misses differences introduced by PCM16 quantization, peak scaling and repeated trimming. Compare original-file scoring with bundled-file scoring, including quiet, over-range and silence-boundary fixtures.
Suggestion — Make the time-box exit operational. The schedule spans roughly eight hours, starts bundle construction before its writer is implemented, and gives boxes six-hour watchdogs that can outlive noon. Define bounded phase cutoffs, a shared absolute rental deadline, and exactly which artifacts and blocker report are produced when time expires.