The library
Rendered from docs/specs/2026-09-26_k-docker-image.md; the markdown file is the record.
On this page
- Deliverable run line (team name Cross Exam, set Sat Sep 26 ~11:10)
- Problem
- Scope change at 06:35 (recorded, not a rewrite)
- Decisions
- Files
- Tests and acceptance criteria
- Doc impact
- Asks to the main chat (relayed via Nathan; their files)
- Results
- Deviations and incidents
- Codex plan review (06:33, docs/reports/2026-09-26k-docker-image-plan-review.md)
- Audits
Run spec: K, the offline CPU Docker image and the end-to-end runner (Sat Sep 26, 2026, ~06:20)
Not a graded deliverable (NSA, Sat Sep 26 ~12:00); kept as reproducibility evidence. The submitted TSV (Sat 12:03 onward) uses fusion_v2 (A3 w0.2 + E, M5 included); this image reproduces the earlier e_on_a rule and is kept as reproducibility evidence only. Since cc6670c (Sat 12:35) the runner's default is fusion_v2, and B6 pins it.
docker/build.shdoes not stage the M5 checkpoint, so a rebuild as-is would produce an image whose runner refuses to start (nomodels/m5_shipped/hashes.json). A rebuild on the new default must first copymodels/m5_xlsr_ft_20260926-0741/modeltodocker/build/models/m5_shipped(+627 MB).
Status: P1 and P2 frozen (deep mode). Plan: docs/reports/2026-09-26_k-docker-image-plan.md.
- Worktree: branch
main,~/Projects/hearsay, shared with the main chat, the D-track chat, the M3 chat, the M5 chat and the oversight chat. Stage only K files by path; nevergit add -A; never push without Nathan. - Handoff:
docs/handoffs/2026-09-26_docker-handoff.md(updated by the oversight chat at ~06:15 with the speech gate and speaker-drift rows). - Rung: K (
docs/plan.mdladder). Gate: first amd64 image with the M1 payload by Sat 14:00. - Time box: plan approved by ~06:50; build and tests until ~09:30; the emulated image build runs in the background. Hard stop 10:00: whatever exists then is reported.
- Decisions at P1 freeze (Nathan, 06:02): install Colima (done: Colima 0.x, vz + Rosetta, 6 CPU / 6 GB, VM disk on the external drive; an amd64
python:3.12-slim-bookwormcontainer reportsx86_64); ship the M1b v3 probemodels/m1_wav2vec2-xls-r-300m_L7_20260926-0521; layer truncation on, gated by an identity test. Delivery to NSA (registry vs tarball) is deferred to the Sunday rebuild.
Deliverable run line (team name Cross Exam, set Sat Sep 26 ~11:10)
docker run --network none \
-v <test_dir>:/data:ro -v <out_dir>:/out \
-v <path>/HearsayScoreKey4TeamX.tsv:/tmpl/key.tsv:ro -e HEARSAY_TEMPLATE=/tmpl/key.tsv \
-e HEARSAY_TEAM=CrossExam hearsay:20260926-0916
It writes <out_dir>/CrossExam_predictions.tsv (header filename<TAB>cm-score, 1.0 = synthetic, rows in the template's order). The runner's own default team name is HEARSAY, which is wrong for the deliverable, so HEARSAY_TEAM=CrossExam must always be passed; no rebuild is needed because it is read at run time. The template mount is optional when the key file sits directly under <test_dir>; the entrypoint then finds it.
Problem
NSA's submission requires a Docker image that "executes inference on the provided test set without major configuration changes". Nothing exists: no Dockerfile, no .dockerignore, no docker/, and no end-to-end runner. Only M1 goes from audio to TSV today (scripts/make_probe_csv.py), and it writes into submissions/ and appends submissions/log.csv, which is wrong inside a container. The image must reproduce the logged M1b v3 TSV (submissions/20260926-0522_M1_m1_wav2vec2-xls-r-300m_L7_20260926-0521.tsv, holdout minDCF 0.0717, EER 0.014, In-the-Wild minDCF 0.342) on CPU, on amd64, with no network, and be ready to take the fusion rule the main chat freezes at 22:00 without a redesign.
Three facts found during P1/P2 shape the design:
- The shipped probe's
segmentflag is wrong.scripts/train_probe.py:81builds the metadata path from the comma-joined train namensa_train_sample_v3,asv19_addon_v3, which does not exist, soprobe.segmentsilently staysFalse(the 0518 probe, one train set, readsTrue). The logged TSV is correct becausescripts/tsv_from_embeddings.pyused precomputed segment-mode embeddings. The runner therefore never trusts the flag (D5). - The lockfile's torch 2.14.0 pulls CUDA on Linux:
cuda-toolkit,nvidia-cudnn-cu13,tritonand 43nvidia-*packages undersys_platform == 'linux'markers. The image must install the CPU pair from the PyTorch CPU index and keep every other locked version, because the probe is a joblib pickle of a scikit-learn 1.9.1 pipeline (D1). models/hc_selectedis a symlink the D-track chat retargets (05:54:hc_lgbm_20260926-055451, v5). The build resolves it at build time and records what shipped (D3).
Scope change at 06:35 (recorded, not a rewrite)
While this plan was being written the main chat shipped its own end-to-end runner: scripts/run_pipeline.py and src/hearsay/pipeline.py (audio → every registered engineered detector plus the two deep scorers → the persisted fusion constants in models/fusion_v0/constants.json written by scripts/fuse.py (commit 22992c4) → the speech gate → one AnalyzeResponse JSON per file, a resumable results.jsonl, timings.csv, run_meta.json and the TSV; --compare-tsv for parity). It reproduces the export paths exactly, including the float16 rounding of the stored embeddings, and it ignores the probe's wrong segment flag. Fusion and the TSV are that chat's lane, so D4–D11 below are superseded: the image ships their runner as its entrypoint, and the K deliverable is the image, its scripts, its checks and its tests (D1–D3, D12–D14). The runner I had written was dropped unstaged; its design points that still apply to their runner (a per-detector cache header keyed on model identity, offline enforcement at start, an 8-module encoder truncation for the layer-7 probe, a tab-only template reader) are listed under "Asks to the main chat". Their CLI: --in --out --template --team --limit --rule --threads --constants --probe --hc --preflight --no-tsv --compare-tsv, with HEARSAY_TEAM (default HEARSAY), HEARSAY_TEMPLATE and HEARSAY_RULE read from the environment.
Decisions
- D1 Dependencies from the lock, CPU torch. Base
python:3.12-slim-bookworm;apt-get install --no-install-recommends ffmpegin one cleaned layer. Requirements are exported at build time fromuv.lock(uv export --frozen --no-dev --no-hashes --no-emit-project --no-emit-package torch --no-emit-package torchaudio --no-emit-package lightgbm), filtered of lines starting withnvidia-,cuda-andtriton, and installed into/opt/venv; thentorch==2.14.0andtorchaudio==2.11.0fromhttps://download.pytorch.org/whl/cpu.lightgbmis never installed (architecture invariant 7).src/hearsayis installed editable (--no-deps -e .);PYTHONPATH=/app/srcis the fallback if the build backend is unavailable. The exact RUN block is fixed by the toolchain probe in the plan. - D2 Offline.
ENV HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_HUB_DISABLE_TELEMETRY=1. The entrypoint passes--require-offline, so a container started without those variables refuses to run. Every image check runs with--network none. - D3 Shipped assets and identity.
COPYweights/wav2vec2-xls-r-300mandweights/Spectra-AASISTto/app/weights/<same name>(the pathshearsay.embed.load_backboneandhearsay.spectra.WEIGHTSderive fromREPO);weights/spkrec-ecapa-voxceleb(85 MB) likewise, because the runner registers speaker drift as evidence; the shipped bundles are staged bydocker/build.shintodocker/build/models/with symlinks resolved and copied to/app/models/: the probe under its real name (hearsay.pipeline.PROBE_DIR),hc_selected,cmp_selected(the compression detector's bundle) andfusion_v0/constants.json(as amended 06:35).docker/build.shresolvesPROBE_DIRandHC_DIRwithreadlink -finto build args and writes/app/BUILD_INFO(git sha, dirty flag, stamp, probe dir, hc dir).docker/assets.py freezewrites/app/assets.json(sha256 and size of every shipped model file and of the backbone weight files); the entrypoint verifies it before handing over to the runner (the build always writes the manifest; the check is skipped only if it is absent)..dockerignoreexcludes.git,.venv,data/,outputs/,submissions/,models/(staged copies are used instead),tests/,docs/,.cache,weights/wavlm-*; the*.wav-style lines only cover root-level files in Docker's semantics, and the audio trees are excluded wholesale throughdata/andtests/. - D4 The runner,
scripts/run_pipeline.py. Also runs on the Mac (that is how Mac-side parity is measured). CLI:--data DIR --out DIR [--template TSV] [--team NAME] [--detectors m1b|m1b,spectra,handcrafted] [--fusion bundle.json] [--policy speech_gate|none] [--limit N] [--device cpu] [--m1-mode segment|windows|auto] [--no-truncate] [--require-offline] [--app-root DIR]; environmentHEARSAY_TEAM,HEARSAY_DETECTORS,HEARSAY_TEMPLATE,HEARSAY_FUSION,HEARSAY_POLICY,OMP_NUM_THREADS. Flow: environment checks (offline, ffmpeg on PATH, threads) → list files (D4a) → asset verification → per detector, in order: load, preflight (silence and chord through that detector, finite and reproducible, values recorded), score every file not in the cache withscore_filessemantics, append to the cache, unload before the next → fusion (D8) → speech-gate policy (D9) → TSV (D10) → sidecar → summary → exit.- D4a Listing. With a template: read it as tab-separated (
filenamecolumn), resolve each name by basename against a recursive index of/data(nested layouts work; duplicate basenames in different directories fail as ambiguous), keep template order, and keep a row for a name that is absent from/data(fallback score, flagmissing_file, warning). Without a template:hearsay.submission.list_test_files(sorted recursive listing ofAUDIO_EXT). Duplicate template names fail before scoring. An empty listing fails with no TSV.--limit Nkeeps the first N rows and suffixes the output name with-limitN. Each file's sha256 is computed at listing time for the cache key.
- D4a Listing. With a template: read it as tab-separated (
- D5 M1 scoring.
load_backbone(probe.backbone, "cpu")→truncate_backbone(model, probe.layer)keepsencoder.layers[:layer+1](verified bit-identical forhidden_states[layer]under transformers 5.17, which does not tie the last hidden state for Wav2Vec2; keepinglayer+1is also correct under the v4 semantics) → per fileprepare_segment(x)thenembed_segment(model, ·)→ cached logitprobe.llr(e)[0](prior-neutral LLR, the same quantityscripts/export_probe_scores.pyexports) → posteriorsigmoid(llr + log(π/(1−π))), π = 0.3, no cost shift (identical toscripts/make_probe_csv.py:55-67). Fallback score is the prior-only posterior, 0.3.--m1-modedefaults tosegment; a probe whose flag disagrees prints a warning; onlyautofollows the flag.torch.manual_seed(0); threads fromOMP_NUM_THREADS, default all cores. - D6 Spectra scoring (shipped, not in the default detector list until fusion says so).
hearsay.spectra.load_spectra("cpu"); per fileprepare_input(x)(band match, no crop) →score_clip(model, x, ("zero",), 16)→ logitsynth_logit(out["zero"]), scoresigmoid(logit); peak normalisation off. These are the settings of the published export (outputs/detector_scores/spectra_aasist_summary.json: pad_mode zero, peak_norm false); the constants live in one block and a test cross-checks them against that file when present. - D7 Handcrafted scoring (shipped, same status).
HandcraftedDetector(model_dir=<app>/models/hc_selected)throughsafe_run(det, ClipContext(path)); logit is thehc_logitfeature,log(p/(1−p))with p clipped at 1e-6, the same asscripts/train_handcrafted.py:decision; a non-okresult gives the fallback plus a flag. - D8 Fusion bundle. A JSON file
{"detectors": [...], "standardize": {name: {"mu", "sd"}}, "rule": "identity" | "zmean" | {"type": "linear", "weights": {name: w}, "intercept": b}, "platt": [a, b], "pi_synth": 0.3, "source": "..."}.apply_fusionstandardises each detector's logit with its inner-OOF mean and sd, applies the rule, thenp = sigmoid(a·fused + b + log(π/(1−π))), which is the arithmetic ofscripts/fuse.py(standardisation,zmean,stack_nonlj, the Platt map and prior shift). With no bundle the rule is identity on M1b, and the output equals D5 exactly. A file missing any detector logit under a bundle gets the fallback plusfusion_missing:<det>; a bundle naming a detector the runner cannot produce fails before scoring. This is the contract forfuse.py --writeto fill. - D9 Speech-gate policy (handoff, 06:15). The runner runs
hearsay.detectors.speech_gateper file as a cheap CPU pass, recordsis_speechand the reasons in the sidecar, and appliesapply_default_answer(fused, is_speech)after fusion: gated files get 0.02 plus 1e-4 times their fused score. A decode failure is gated the same way (is_speech false). Default--policy speech_gate;nonedisables it for parity against the logged M1 TSV, which predates the gate. The preflight silence and chord therefore land near 0.02 in the final scores; the per-detector preflight values are still recorded raw. - D10 Output.
write_submissionto/out/<TEAM>_predictions.tsv(HEARSAY_TEAM, placeholderteamName); if it exists,<TEAM>_predictions-<stamp>.tsv; never overwrite, neverappend_loginside the container (log.csv is a repo artifact; Nathan logs a container TSV on the Mac once parity passes). Beside it,<stem>.sidecar.csv(per file: id, score, flag, per-detector logits, is_speech) and<stem>.run.json(git sha and dirty flag, BUILD_INFO, library versions, ffmpeg banner, asset hashes, detectors, fusion rule, policy, truncation depth, preflight values, wall time per detector, flag counts, share above 0.5,partialfor--limit). Scores outside [0, 1] are caught at scoring time (fallback, flagout_of_range), never clipped. Exit code 0 whenever a TSV was written, even with flagged rows; non-zero only when none was. - D11 Cache.
/out/cache/<detector>.jsonl: a header line{manifest_sha, config_sha, detector, created}then one line per file{id, sha256, logit, score, flag, seconds}. A different manifest hash (file set and content) or config hash (probe sha, truncation, mode, pad mode, bundle) discards the cache with a notice; a corrupt trailing line is skipped and rescored; rows are looked up by id and sha256. - D12 docker/ scripts (as amended 06:35).
entrypoint.sh(set -euo pipefail; threads from the cgroup quota intoOMP_NUM_THREADS/MKL_NUM_THREADS; a lone/data/*.tsvbecomesHEARSAY_TEMPLATE;assets.py verifybefore anything runs; thenexec python scripts/run_pipeline.py --in /data --out /out --threads N --require-offline "$@"with N capped at six);build.sh(docker buildx build --platform linux/amd64 --load, build args, tagshearsay:<stamp>andhearsay:latest,BUILD_INFO);smoke.sh(3 files: WAV, MP3, FLAC; reversed template;--network none; in-image self-checks; row count and order check; run twice and compared within 1e-6);parity.py(pcm-hash/pcm-diff: decoded-PCM hashes viascripts/hash_audio.hash_one; score parity is the runner's own--compare-tsv);assets.py(freeze,verify). - D13 Tests are hermetic (as amended 06:35).
tests/test_docker_image.py: B1–B10 on the Dockerfile,.dockerignore(B4/B5 are name and prefix checks on the short, reviewed ignore file rather than the pattern matcher the plan sketched; the in-image self-check insmoke.shis what proves the weights and model paths actually exist in the image), the scripts, the staging resolution and the asset manifest, P2–P4 onparity.py,smoke.shandbuild.sh, plus a static check that the entrypoint's flags exist in the main chat's runner. The runner's own tests are the main chat's. Image checks are the scripts in D12, run by hand and recorded below. - D14 Out of scope for this rung. M5 in the image; running speaker drift in the pipeline (weights and dependency ship, the detector is evidence only); the explanation report; registry push or tarball; the Sunday rebuild (a separate execution that drops in the fusion bundle and rebuilds).
Files
Create: Dockerfile, .dockerignore, docker/entrypoint.sh, docker/build.sh, docker/smoke.sh, docker/parity.py, docker/assets.py, docker/.gitignore, tests/test_docker_image.py, this spec, the plan file. (scripts/run_pipeline.py and tests/test_docker_runner.py were dropped at 06:35: the runner is the main chat's.)
Modify: README.md (Docker section). Conditional (other chats hold uncommitted edits; stage only my hunk): docs/STATUS.md Docker row, CLAUDE.md AI-use disclosure bullet, docs/architecture.md section 10 item 7.
Never: src/hearsay/embed.py, probe.py, submission.py, spectra.py, m5_*.py; scripts/make_probe_csv.py, train_probe.py, extract_embeddings.py, fuse.py, score_spectra.py, m5_*.py; submissions/; splits/; the detector modules; pyproject.toml; uv.lock.
Tests and acceptance criteria
Test IDs are the P2 inventory: B1–B13 (build, assets, truncation), R1–R5 (runtime environment), L1–L9 (listing), D1–D5 (decode and preflight), M1–M8 (M1 scorer), F1–F7 (other scorers and fusion), G1–G4 (speech-gate policy: gated file gets 0.02 + 1e-4·fused, speech file untouched, decode failure gated, --policy none leaves scores as fused), C1–C5 (cache), O1–O8 (output), P1–P4 (docker/ scripts). Historical bug classes mapped: silent default on missing metadata (M3), tiling seam (M2), resume mixing settings (C3), stale code on the box (R4, B10), TSV format and no-overwrite (O1–O3, L7), decode failure losing a row (D1, L5), MPS vs CPU drift (T7, P2), symlinked model dir (B7), CUDA wheels on Linux (B3), LightGBM at inference (B9), tab-separated template read with a comma reader (L2), sponsor polarity (M1).
| ID | Command | Pass |
|---|---|---|
| T1 | uv run pytest -q tests/test_docker_image.py | all pass, hermetic, under a minute |
| T2 | uv run pytest -q and uv run ruff check . | the pre-existing suite plus the new tests pass; ruff clean |
| T3 | uv run pytest -q tests/test_pipeline.py -k truncated_backbone (the main chat's needs_weights identity test for the 8-module truncation) | passes on this Mac |
| T4 | Mac-side CPU run of scripts/run_pipeline.py --in data/nsa/HackGTHearsayTesting --template data/nsa/HearsayScoreKey4TeamX.tsv --limit 50 --compare-tsv submissions/20260926-0602_M4_fusion_zmean_m1bv3_hcv5_m3.tsv | Spearman > 0.99, max abs diff < 0.05 |
| T5 | docker/build.sh | amd64 image; platform.machine() inside is x86_64; size recorded (unpacked filesystem under 6 GB; the image store's own figures noted beside it) |
| T6 | docker/smoke.sh | 3-row TSV in reversed template order, --network none, exit 0, preflight finite and reproducible, repeat runs agree within 1e-6, the in-image self-checks (paths, numba, no lightgbm, manifest, speaker drift offline) pass, and a container started with an offline variable unset is refused with no TSV written (D2) |
| T7 | 50 template files through the image with --network none, the runner's --compare-tsv against the logged zmean TSV, then parity.py pcm-hash/pcm-diff against the Mac | Spearman > 0.99, max abs diff < 0.05; PCM hashes identical for all 50 |
| T8 | full 1,671 files through the image, background | wall time recorded; if unfinished at the hard stop, the extrapolation and the blocker are recorded; the host-side append_log action is named |
| T9 | uv run pytest -q tests/test_docs_consistency.py after the README edit | passes |
Regression: any previously passing test failing, any change to a never-touch file, any submission TSV changed, or T4/T7 outside threshold.
HEARSAY standing failure modes: (1) laundering/telephony/replay: waived, nothing is modelled; PCM parity is measured (T7) and the smoke set includes MP3 and FLAC; (2) generator/speaker overfitting: waived, nothing is fit, validation numbers are quoted from the log; (3) class prior: π = 0.3 applied exactly as the M1 scripts do, no thresholding, share above 0.5 recorded; (4) shortcuts: waived on modelling; the runner never passes filenames or timestamps to a scorer; (5) loader format errors: pinned by the existing suite; T6 exercises three formats inside the image.
Doc impact
Committed: this run spec; the plan file and its Codex review; README.md Docker section (build, run, env vars, mounts, outputs, smoke and parity); the CLAUDE.md AI-use disclosure bullet for the K chat. Deferred by agreement (07:05): docs/STATUS.md Docker row and docs/architecture.md section 10 item 7 are written by the oversight chat from the numbers this spec reports (it asked for the in-image parity and the image size and owns those two docs today); my drafts of both were withdrawn from the working tree so the two chats do not edit the same lines. Not touched: docs/plan.md (design unchanged), docs/code-map.md (oversight chat), scripts/train_probe.py (main chat).
Asks to the main chat (relayed via Nathan; their files)
- Done by the main chat (541afdb):
scripts/train_probe.pyreads each set'sextract_meta.json; the 0521 probe artifact was patched tosegment=True. - Done by the main chat (1394ca0):
hearsay.pipeline.truncate_backbonekeepsencoder.layers[:probe.layer + 1]by default (--no-truncatekeeps 24) andtests/test_pipeline.py::test_truncated_backbone_keeps_layer_7_identical(needs_weights) pins bit-identity on the real backbone; it ran and passed on this Mac in the 419. - Done by the main chat (1394ca0):
results.jsonlcarries a header line with the model identities (git sha, probe, hc, constants sha, scorers, mode, truncation); a mismatch moves the old cache aside;--freshforces a rescore. - Done by the main chat (1394ca0): the template parse is tab-only and fails loudly.
Results
| ID | Outcome |
|---|---|
| T1 | tests/test_docker_image.py: 16 passed in 10 s (hermetic) |
| T2 | uv run pytest -q: 419 passed at 07:27 (the suite grew from 301 to 419 during this rung as other chats landed tests; 16 of them are tests/test_docker_image.py), uv run ruff check . clean |
| T3 | the equivalent test is the main chat's tests/test_pipeline.py::test_truncated_backbone_keeps_layer_7_identical (needs_weights, real XLS-R): ran and passed on this Mac in the 419 |
| T4 | Mac CPU, main chat's runner, 50 template files, rule zmean, gate on, against submissions/20260926-0602_M4_fusion_zmean_m1bv3_hcv5_m3.tsv. First run (06:42, runner as of 06:34, 24 layers): Spearman 1.0, max abs diff 2.4e-5, mean 2.3e-6, 0 rows over 0.01; 0 gated, 0 decode errors, share > 0.5 = 0.22; 4.74 s per file on 6 threads. Rerun (07:42, final runner c0f4022 with truncation): the same parity numbers to the digit (Spearman 1.0, max 2.39e-5, mean 2.30e-6), 0 gated, 0.745 s per file, 49.6 s wall. Preflight silence 0.0201 / chord 0.0201 (fused before the gate 0.985 / 0.999). |
| T5 | Six builds on Colima (Rosetta): builds 1 and 2 (06:37, 06:48) found the two image bugs and the preflight jitter; builds 3 to 5 died in the disk incidents; build 6 (hearsay:20260926-0719, 07:19 to 07:22 from scratch, runner c0f4022) carried the first full set of passing checks but predated the final entrypoint (Codex audit round 1); build 7 is the record: hearsay:20260926-0753 = hearsay:latest, from scratch in about 3.5 min (07:53 to 07:56), source 0963ed8 (entrypoint with --require-offline and the six-thread cap; runner c0f4022 (includes the 1e-6 preflight tolerance, the cache identity header, truncation on by default, the patched 0521 probe with segment=True, the offline ECAPA fix), ECAPA dir shipped untouched). amd64; docker images lists 7.5 GB, docker image inspect .Size 2.66 GB (compressed content), unpacked filesystem about 4.5 GB (2.6 GB weights, 1.3 GB venv, 461 MB system libraries, 9.4 MB models). BUILD_INFO sha=0963ed8 dirty stamp=20260926-0753 probe=m1_wav2vec2-xls-r-300m_L7_20260926-0521 hc=hc_lgbm_20260926-055451 cmp=cmp_lgbm_20260926-0233 fusion=fusion_v0. A rebuild with cached layers is expected to take about 2 min (not yet measured after the cache-keeping change; the Sunday rebuild will record it); build.sh keeps the builder cache (only old tags and dangling images are pruned). |
| T6 | Build 6 (hearsay:20260926-0719 = latest, runner c0f4022): in-image self-checks pass (x86_64, torch 2.14.0+cpu, no CUDA, hearsay.embed.REPO = /app with the weights and the runner's PROBE_DIR/HC_DIR/CONSTANTS_PATH present, librosa/numba JIT, no lightgbm, 70-file manifest verified with full hashes, ffmpeg 5.1.9, speaker drift loads ECAPA offline and returns "one consistent voice" on a noise clip). Three files (WAV, MP3, FLAC) with a reversed template through --network none: 4-line TSV in the template's order, exit 0, all three gated as non-speech (correct: a sine, pink noise and a tone), preflight silence and chord 0.0201 (0.985 and 0.999 before the gate), 52 s wall for the run including model load; the second run's TSV is byte-identical (max diff 0.0). History: build 1 failed on my manifest frozen inside models/ and a missing parity.py; build 2 hit the runner's exact-equality preflight (the first forward pass after the engineered detectors differs from later passes: thread-pool warm-up, OMP_NUM_THREADS=1 is exact throughout, MKL settings do not help; the main chat added a 1e-6 tolerance) and speaker_drift reading label_encoder.txt from the Mac's HF cache (fixed by the D-track chat in 1cc4fbe); builds 3 to 5 were casualties of the disk incidents; build 6 is the record. My smoke script's last check read a n_decode_error key the final runner no longer writes; fixed and re-run at 07:31: exit 0, 42 s wall, 3 gated, 0 decode errors, two runs identical. Build 7 (hearsay:20260926-0753, the final entrypoint), 07:56: all self-checks pass again; the new negative check passes: a container started with TRANSFORMERS_OFFLINE=0 is refused by --require-offline and writes no TSV; the three-file runs give the same TSV as build 6, exit 0, two runs byte-identical, preflight 0.0201/0.0201; run_meta.json of the parity run records require_offline: true. |
| T7 | 50 template files inside the image, --network none, rule zmean, gate on, --compare-tsv against submissions/20260926-0602_M4_fusion_zmean_m1bv3_hcv5_m3.tsv: Spearman 1.0, max abs diff 1.43e-4, mean 1.26e-5, 0 rows over 0.01 (threshold 0.05); 0 gated, 0 flags, share > 0.5 = 0.22; 4.28 s per file, 257 s wall (the Mac: 2.4e-5 max diff, the x86 float path differs slightly and the float16 rounding of the embedding plus zmean's Platt slope amplify it). PCM: 50/50 decoded-audio hashes identical between the Mac (ffmpeg 7.1.1) and the image (ffmpeg 5.1.9). Two files from the oversight chat's own earlier in-image run: 1.4e-4 and 8e-8. Build 7 re-run (08:00): identical numbers (Spearman 1.0, max abs diff 1.4267e-4, mean 1.2565e-5, 0 gated), 4.02 s per file, 234 s wall, PCM 50/50 identical. |
| T8 | Started 07:33 in the background (after the smoke re-run) on build 6, whose runner is byte-for-byte the one in build 7 (only the entrypoint differs): all 1,671 template files inside the image under --network none with --compare-tsv against the logged zmean TSV; at 4.28 s per file about 2 h under Rosetta (the runner's own Mac figure with truncation is 0.77 s per file, so a native amd64 box should take about 20 to 30 min). Memory during the run: 3.3 GiB resident with every model loaded (docker stats, 6 threads). Disposition of the first attempt: killed at 07:58 with exit 137 (SIGKILL) after 308 files and 24 min 35 s; no full TSV exists from it. The cause is not established by the exit code alone; the likely one is memory: build 7 ran in the same 6 GB VM from 07:53 to 07:56 beside the 3.3 GiB scoring container, and at the same time the VM's 20 GB disk cap was reached (two 7.5 GB image tags plus 8 GB of build cache; the cap held, the external drive kept 11 GB free). Lesson: never build while a scoring run is in the VM. Replacement run: the builder cache was pruned (4.7 GB free in the VM) and the run restarted at 08:05 on the same image hearsay:20260926-0719, resuming from the 308 cached rows (the runner printed 308 cached; its identity header matched); expected to finish about 09:35. Outcome (08:45): stopped by decision, recorded as the permitted incomplete outcome. Rows scored: about 650 of 1,671 (308 in the first attempt, 24 min 35 s, 4.8 s per file; the rest between 08:05 and 08:45 at about 6.5 s per file while builds and audits shared the Mac; the 50-file runs with the VM alone measured 4.0 to 4.3 s per file). Extrapolated full set in this emulation: about 1.9 to 2.0 h with the VM alone, about 3 h shared, so completion by the 10:00 hard stop was no longer possible, and the run used fusion_v0 zmean, retired as the shipped rule at 08:13. It was stopped to free the VM for the rule-change rebuild (building beside it is what killed the first attempt). No full container-made TSV exists, so the host-side append_log action above stays deferred; the blocker is named. A full-set run on the final image with the shipped rule e_on_a is the follow-up if the VM is free for two hours before the Sunday rebuild. Result: addendum below when it lands. If it has not finished by the 10:00 hard stop, that is the named blocker for logging a container-made full TSV; the host-side action once it exists (main chat's submissions/, so run by Nathan or the main chat, never by this chat): uv run python -c "from hearsay.submission import append_log; append_log('K', 'outputs/docker/k-full/teamName_predictions.tsv', validation_score=0.0, notes='in-image full run of the zmean fusion (holdout minDCF 0.0 EER 0.0 per the 06:02 log row); parity vs submissions/20260926-0602_M4_fusion_zmean_m1bv3_hcv5_m3.tsv: <spearman>, max abs diff <d>; image hearsay:20260926-0719 sha c0f4022')" after copying the TSV under submissions/ with a new name. |
| T9 | tests/test_docs_consistency.py: 52 passed after the README edit |
Deviations and incidents
- 07:00–07:09, external drive full (mine). The Colima VM disk was placed on
/Volumes/Crucial P3 NVME Gen 3 2TBat 06:05 (30 GB free then, the internal disk had 16 GB) and grew to 31 GB across three builds until the drive hit 100%; writes inside the VM then failed (the third build died copyingpytorch_model.bin, containerd's blob store returned I/O errors). Other lanes' writes to that drive in that window may have failed; the oversight chat was told at 07:15. Fix: VM deleted (only my images and build cache), drive back to 31 GB free, Colima recreated on the internal disk with--disk 24(07:14). 07:20, second move: one from-scratch build (build 5) took the internal disk from 12 GB to 3 GB free within five minutes (transient footprint of context upload plus layers), so the build was stopped and the VM moved back to the external drive with a hard 20 GB cap (--disk 20, below the drive's 31 GB free, so it cannot fill either disk);docker/build.shnow refuses below 8 GB free on the drive hosting the VM and, after each build, removes old tags and dangling images (the builder cache is kept since ce84cdf, see T5). Build 6 (07:19) and build 7 (07:53) are the ones reported below. Lesson recorded in memory. - Note (Sat 10:30): runner moving again, no rebuild needed. The M4 fusion chat is adding an optional M5 scorer and a
fusion_v2layout to the runner, backward compatible: the default staysfusion_v1/e_on_a, and M5 runs only when a fusion file weights it. The build-9 entry below and the Codex-review item 4 ("the runner always scores all three") describe the image as built at the time and stay as the record. A switch tofusion_v2would needmodels/m5_shippedstaged, the B6 pin moved, and a rebuild with smoke and parity; Nathan decides. - Ownership change at 06:35 (the runner): recorded above.
- Build 9, the shipped-rule image (09:16 to 09:31):
hearsay:20260926-0916=hearsay:latest, source37c26b8(runner v29614c18, default rulee_on_afrommodels/fusion_v1/constants.json; BUILD_INFO listsfusion_v0,fusion_v1,fusion_v2_candidate, all staged, v2 unused because M5 is shelved). Build 8 (08:41) died with "no space left on device" in the VM (three 7.5 GB tags plus cache under the 20 GB cap);build.shnow removes superseded tags before building, with a test. Checks on build 9 with--network none: offline refusal on real input passes; smoke three files exit 0, two runs byte-identical; 50-file parity vssubmissions/20260926-0813_M4_sweep_E_on_A_alpha0.2_our_direction.tsv: Spearman 1.0, max abs diff 2.02e-4 (recomputed independently from the TSVs), mean 9.1e-6, 0 gated, share > 0.5 = 0.18,rule: e_on_a,require_offline: true(the container mount was still named/ref/zmean.tsvinoutputs/docker/image_checks9.log, a leftover name; the file mounted there was the 0813e_on_aTSV, which the independent recomputation confirms; the check script now mounts it as/ref/reference.tsv); 6.98 s per file (the Mac was busy; 4.0 s on an idle VM); PCM 50/50 identical. The first--flipsmoke assertion failed because it demanded a full rank reversal and all three smoke files are gated: the runner pins gated files below 0.001 in both polarities, so only determinate rows can reverse. The assertion now checks reversal on determinate rows and identical gating in both files; keeping the gated block at the minimum in the flipped file is intended (oversight chat, 09:35;docs/consults/2026-09-26_fusion-strategy_RESPONSE.mditem 5: the bottom avoids the 4x error under both readings for a block spoof rate between 0.097 and 0.80). Smoke re-run (09:32 to 09:40): exit 0; self-checks, 72-file manifest with full hashes, offline refusal, byte-identical repeat runs, and the--fliptwin (same order, determinate rows reversed, gating identical, polarityflippedrecorded) all pass. - Shipped fusion rule changed at 08:13 (main chat,
docs/reports/2026-09-26_fusion-sweep-predeclared.md):models/fusion_v1/constants.json, rule "E on A alpha 0.2" (a rank blend of M1b and handcrafted with Spectra used only to suppress false alarms), replacesfusion_v0zmean as the shipped rule; the runner is being updated by its owner.docker/build.shnow stages everymodels/fusion_*/constants.json(verified:fusion=fusion_v0,fusion_v1in BUILD_INFO), and the parity reference for the next image issubmissions/20260926-0813_M4_sweep_E_on_A_alpha0.2_our_direction.tsv. No rebuild until the oversight chat sends "runner final v2"; the T8 full-set record below was made withfusion_v0zmean and stands as the timing and old-rule parity record. - Two image-side bugs found by the smoke test (manifest frozen inside
models/;parity.pynot copied) and one detector dependency on the HF cache (speaker_drift): recorded in T6.
Codex plan review (06:33, docs/reports/2026-09-26_k-docker-image-plan-review.md)
Eleven findings against the plan as written before the 06:35 scope change. Resolution of each:
- In-image checks would hit the exec-form entrypoint: fixed, every self-check and hash run uses
--entrypoint python(orsh,ffmpeg) and--network none. - Corrupt cache tail not truncated: superseded; the cache is now the main chat's
results.jsonl(load_cachedrops a torn last line and appends after it; passed on as ask 3). - Cache fingerprints omit model identity: superseded, same ask 3; in the image the models are fixed at build time and
assets.jsonpins them. - Bundle detectors checked against supported not selected: was already the selected set; moot after the scope change (the runner always scores all three).
- Bundled identity on the raw logit: was already standardized; moot, the runner reads
constants.jsondirectly. - Numeric bundle validation: moot,
hearsay.pipeline.FusionConstantsowns it. - Parity compare subset semantics and NaN rejection: superseded by the runner's
--compare-tsv(shared filenames, Spearman, max abs diff); T4 and T7 report n = 50 shared rows. - Host-side log row: accepted as a named action, not done here.
submissions/is the main chat's; the T8 result row carries the literalappend_logcommand and the TSV path, and an unfinished full run at the hard stop is the named blocker. - Validate the gate policy on the holdout: out of this rung; the gate is the main chat's detector (architecture 10.4). T4/T7 report the gated count on the test files (0 of 50) and parity is measured with the gate active (no test file is gated, so the comparison against the pre-gate TSV is exact).
- L9/O6/fake-config test contradictions: superseded with the runner tests.
- Time box contingency: fixed by starting the build at 06:37 with the weights layers cached before the code was final; the plan's execution note records the intermediate stops (08:00 commit and critique, 09:00 Codex audit, 09:30 T8 addendum, 10:00 hard stop).
Audits
Claude pre-audit critique (adversarial, the audit rubric). Round 1 on 409e331 (07:40): Overall Needs-work, 14 findings: D2's --require-offline missing from the exec line, build-6 results written into the acceptance table instead of the results table, the plan lacking the intermediate stops it was said to hold, a vacuous COPY-source regex in B4, B4/B5 weaker than the plan's sketch, an or assertion in B10, the append_log action not written down, README thread default and smoke wording stronger than the code, an unmeasured memory claim, build.sh discarding the layer cache it was designed around, a stale ask and D3 wording, DIRTY under pipefail, a blank line in CLAUDE.md. All fixed in ce84cdf. Round 2 on ce84cdf (07:52): Overall Acceptable (pass), every item re-verified in code and artifacts (419 passed, ruff clean, 7 files, no forbidden path), five cosmetic nits (D12 exec-line quote, a stale smoke sentence, GB vs GiB, the T5 size criterion's wording, the cached-rebuild estimate) fixed in the commit after it.
Codex audit, round 1 (07:51, docs/specs/2026-09-26_k-docker-image-audit.md): Overall Fail on one substantive finding: the recorded image hearsay:20260926-0719 was built before ce84cdf, so the smoke and parity runs on record never exercised the final entrypoint (--require-offline, the six-thread cap); the saved run_meta.json shows require_offline: false. Accepted: build 7 from the final source, with the smoke test extended by a negative check (a container started with TRANSFORMERS_OFFLINE=0 must be refused and write no TSV), and smoke, parity and PCM re-run on it (results in the T5–T7 rows, "build 7"). Its documentation finding (acceptance rows and asks still describing the dropped runner and asks already fulfilled) fixed in the same commit. Its verification limitation (no writable temp dir in the sandbox, Docker permission-denied) noted; the 419 and the image runs are on this Mac's record.
Codex audit, round 2 (08:04 to 08:10): Overall Fail: (1) T8 read as "pending" while smoke_then_full.log ended in FULL EXIT 137 (the audit read the spec at 08:04; the kill and the 08:05 restart were recorded at 08:06); the row now carries the disposition above, with the cause stated as likely rather than proven. (2) The negative offline check ran on an empty input directory without HEARSAY_TEAM=smoke, so its "no TSV" assertion could not fail: fixed, it now runs on the three real files with the template and asserts /out stays empty; validated on build 7 by hand (a single container, so the resumed full run is not disturbed), the full smoke.sh re-run is deferred until the full run finishes. (3) parity.py pcm-diff accepted two failed decodes as a match: fixed, a failed decode on either side is reported, with a test. Its sandbox limitation (no writable temp dir, no Docker access) is noted; the artifacts on this Mac carry the evidence.
Codex audit, round 3 (08:08 to 08:11, on 33a3380): Overall Acceptable. Plan adherence, Test coverage, Review compliance, Regression check and Documentation Acceptable; Scope discipline Excellent. Its two minor findings, a one-sided decode failure labelled "both" in parity.py and the incident paragraph still describing the builder-cache prune, fixed in the next commit with a test for the label. It independently re-checked the saved build-7 TSV (50 valid rows in template order, Spearman 1.0, max diff 1.42672e-4) and the 50 PCM records; it notes a repository-wide ruff finding in another lane's uncommitted scripts/m5_train.py, not K's. T8 stays the named open item until the resumed run lands (addendum below).
Codex audit, round 4 (08:12 to 08:14, on 8a682ba): Overall Acceptable, confirmed. All dimensions Acceptable (Scope discipline Excellent); it independently recalculated the build-7 parity (matching ranks, max diff 1.42672e-4) and the 50/50 PCM match, and repository-wide ruff passed again. One cosmetic finding (the smoke.sh header still said repeat runs must be byte-identical) fixed in the commit after it. Open by design: T8's addendum and the full smoke.sh re-run on build 7 (deferred until the full run leaves the VM), both recorded below when they land.