2026-09-26 m4 fusion v2 m5 scorer (plan review)

The library

Rendered from docs/reports/2026-09-26_m4-fusion-v2-m5-scorer-plan-review.md; the markdown file is the record.

Critical — The error tally misses resumed failures and handcrafted failures. Step 4.10 counts errors after scoring, but cached rows take an earlier continue in scripts/run_pipeline.py. A failed run can therefore resume and report zero errors. Additionally, errors are keyed as handcrafted, while the exit check looks up handcrafted_v5. Aggregate errors from the completed docs collection using FUSED_FROM_DETECTOR to normalize names. Test fresh, partially cached, and fully cached failures, including exit code 4 after TSV creation.
Critical — Step 7.3 does not produce or verify both polarities. The runner’s --flip emits only the flipped output; it does not also emit our-direction predictions. The proposed command compares that flipped output against the our-direction candidate. Run the full set without --flip, then reuse its cache with --flip and the flipped reference. Explicitly verify all 1,671 rows and A5’s verdict/E-rule checks before accepting the result.
Critical — The submission-log requirement is explicitly excluded. Step 7 creates new TSVs, but A8 forbids logging them, and the runner deliberately never calls append_log. This conflicts with CLAUDE.md and the requested completion criterion. Add a plan-level step logging the completed full-set artifact with its validation score, clean-only score, and candidate status; logging need not change the shipped selection. If blocked, name the blocker explicitly.
Critical — API scorer discovery assumes a field the rank constants do not contain. Step 8 describes reading “the file’s detectors” with json.loads, but neither the existing v1 writer nor the proposed v2 writer emits that field. FusionConstants derives it from the rule layout. Have file_scorers reuse FusionConstants.load(path).detectors, with the proposed memoization, and test actual writer-shaped v1/v2 JSON. Also reconcile the missing-file test: the stated env-derived fallback has three scorers, whereas the proposed assertion expects the expanded four-element DETECTOR_ORDER.
Critical — The reported outer holdout is used for model selection. fuse_sweep_m5.py filters candidates using holdout_brief and chooses the winner using ITW; the spec explicitly calls this selection on the held-out sets. Predeclaring that procedure does not preserve an untouched outer evaluation. Keep this implementation limited to reproducing the frozen candidate, label these numbers as selection results, and identify the independent evaluation still pending. Any renewed selection needs an inner-only procedure with an untouched outer holdout.
Critical — The schedule does not enforce its stated completion boundary. The heading ends at 15:00, the table ends at 15:45, and the critique loop has no iteration or deadline limit. Passing the two early stops permits indefinite continuation, while A5 can remain unfinished in the background. Set one final deadline covering full-set verification, documentation, and bounded audit iterations; specify the report and fallback if any required gate remains incomplete.
Suggestion — Reject non-finite and malformed constants explicitly. The proposed weight checks accept NaN: both v <= 0 and abs(sum(weights.values()) - 1) > tolerance evaluate false for it. Reference validation also accepts singleton NaN arrays and some multidimensional arrays. Require finite weights and finite, one-dimensional, nonempty rank references before checking positivity, normalization, and ordering; extend T2 accordingly.
Suggestion — The relocated scorer notification cannot detect omitted flags. Step 4 retains resolve_scorers returning the maximal DETECTOR_ORDER, then stores that result as requested. Consequently, set(scorers) - set(requested) is always empty for fusion runs, including default flags that omit M5. Preserve the actual parsed flag selection separately so the notification reflects what the user requested.