BD6.8F — runtime determinism diagnostic, the surprise is in production (2026-05-02)

TL;DR — the question was "does greedy decode fix the 4 BD6.5 holdouts?". The answer is split (2 yes, 2 no). The bigger discovery: PRODUCTION BD6 pass-1 collapses from 19/19 to 1/19 under pure greedy. anchor_positive.jsonl was captured WITH rep_penalty=1.15 + no_repeat_ngram=2 + cuda_rep_penalty=1.05 in the loop, so the gate's "19/19" is a co-engineered weights+decoder measurement, not a pure-weights one. BD6.5 weights are actually MORE robust under greedy (16/19) than the production pack (1/19). This reframes the entire BD6.x cycle: "weights vs decoder" is a tangled diagnosis, and surgery alone cannot win without considering decoder co-design.

What was tested

No training. No LoRA. No merge. No architecture change. Just four runtime configurations, each running the same 19 anchor prompts via the production anchor_eval.py harness:

                       PHYS05 PACK            sampler spec for phys05_code_skeleton
                  ┌─────────────────────┬─────────────────────────────────────┐
A) prod / current │ BD6 pass-1 (.planck)│ rep_pen=1.15 ngram=2 cuda_rep=1.05  │
B) prod / greedy  │ BD6 pass-1 (.planck)│ rep_pen=1.00 ngram=0 cuda_rep=1.00  │
C) v5   / current │ BD6.5      (.planck)│ rep_pen=1.15 ngram=2 cuda_rep=1.05  │
D) v5   / greedy  │ BD6.5      (.planck)│ rep_pen=1.00 ngram=0 cuda_rep=1.00  │
                  └─────────────────────┴─────────────────────────────────────┘

temperature is already 0 in production (greedy in the strict sense = argmax). The "current" and "greedy" labels above differ ONLY in repetition-penalty + ngram-blocker.

Headline results

| cell | pack | spec | anchor pass | rate | |------|------|------|-------------|------| | A | production BD6 pass-1 | rep=1.15, ng=2 (current) | 19/19 | 100.0 % | | B | production BD6 pass-1 | rep=1.00, ng=0 (greedy) | 1 /19 | 5.3 % | | C | BD6.5 (rejected v5) | rep=1.15, ng=2 (current) | 15/19 | 78.9 % | | D | BD6.5 (rejected v5) | rep=1.00, ng=0 (greedy) | 16/19 | 84.2 % |

Per-row truth table (only the 4 BD6.5 holdouts and the moves)

| task_id | tgt len | A prod/cur | B prod/greedy | C v5/cur | D v5/greedy | verdict | |----------------|---------|------------|----------------|----------|-------------|---------| | MBPP/53 | 256 | ✓ | ✗ | ✗ holdout | ✗ still fail | weights — both decode modes can't recover | | HumanEval/34 | 182 | ✓ | ✗ | ✗ holdout | ✓ greedy fixed | DECODER noise — recoverable | | HumanEval/45 | 249 | ✓ | ✗ | ✗ holdout | ✓ greedy fixed | DECODER noise — recoverable | | HumanEval/85 | 366 | ✓ | ✗ | ✗ holdout | ✗ still fail | weights — both decode modes can't recover | | MBPP/20 | 66 | ✓ | ✗ | ✓ | ✗ greedy lost | rep-penalty HELPS this short prompt | | HumanEval/53 | 101 | ✓ | ✓ | ✓ | ✓ | invariant under both modes |

The 4 holdouts split exactly half: HE/34 and HE/45 are decoder-noise holdouts, recoverable by removing rep-penalty. MBPP/53 and HE/85 are real weight-gap holdouts, robust to decoder change.

The production-fragility surprise

A → B drops 19/19 → 1/19 (only HE/53 survives pure greedy on the production pack). This is much more than a quality dip — BD6 pass-1 is essentially unable to produce usable Python without the repetition-penalty + ngram-blocker active.

That's because:

anchor_positive.jsonl (the 19-row positive target set used by every

BD6.x training pass) was captured by running the production pack with the production sampler (rep=1.15, ng=2) and recording the outputs. The "anchor 19/19" measurement that has gated every BD6.x pass is a function of WEIGHTS × DECODER, not weights alone.

Without the penalty, the BD6 pass-1 base distribution falls into

loops/repetition on Python keywords (return, def, , etc.) and the verifier rejects almost everything.

Implication: the strict gate "anchor 19/19" was always testing the joint weights+decoder system, never weights in isolation. Calling a pack "production-ready" means "ready under THIS decoder config." Any proposed runtime decoder change has to re-run the gate from scratch.

What this means for the BD6.5 holdouts

Per the user's decision tree:

If deterministic fixes holdouts: next = decode/extractor hardening

If deterministic does not fix: next = BD6.8D token-weighted CE on BD6.5 dataset shape

The answer is both, partitioned:

HE/34, HE/45 — fixable by decoder change. Removing rep-penalty

on phys05_code_skeleton recovers them. But that same change drops production to 1/19, so we cannot ship that decoder. The path to fixing these without breaking production is to find an intermediate decoder spec (e.g., rep=1.05 + ngram=0, or rep=1.15 + ngram=0, etc.) — a scan in the (rep_penalty, ngram) hyperparameter space that keeps prod at 19/19 while letting v5 reach 18-19/19. This is Lever F+ — runtime hyperparameter scan, not surgery.

MBPP/53 (256 chars), HE/85 (366 chars) — not fixable by

decoder. Both fail under greedy on v5 too. These are real weight gaps. This is where Lever D (token-weighted CE) is still on the table. The diagnostic from BD6.5/BD6.6 stands: long-target loss dominates gradient and the LoRA "tries hard" early then drifts under poison pressure. Token-weighted CE addresses exactly that.

MBPP/20 — surprise: rep-penalty helps this short prompt (66

chars). Removing it loses the row. This means the (rep_penalty, ngram) choice has both anchoring and de-anchoring effects depending on row content, which makes scan-only Lever F+ a delicate problem.

The cleaner reading of the BD6.x trajectory

The history:

BD6 pass-1   anchor 19/19   under (rep=1.15, ng=2)        — production
BD6.3        anchor  0/19   under (rep=1.15, ng=2)        — REVERT
BD6.4        anchor  7/19   under (rep=1.15, ng=2)        — REVERT
BD6.5        anchor 15/19   under (rep=1.15, ng=2)        — REVERT (15/19 < 19/19)
BD6.5        anchor 16/19   under (rep=1.00, ng=0)        — JUST MEASURED, reframes BD6.5
BD6.6        anchor 11/19   under (rep=1.15, ng=2)        — REVERT
BD6.7a-c     anchor 12/19, 10/19, 12/19   under (rep=1.15, ng=2)  — REVERT (KL ladder fail)

BD6.5 was 16/19 under greedy — closer to the 19/19 ceiling than any of the post-replication or KL attempts. The lossy 15/19 measurement we gated on was inflated/diminished by decoder co-engineering. BD6.5 is even better than we thought as a weight artifact. The reason it failed the gate isn't that its weights are 15/19 — it's that the gate's "production decoder" config was tuned for production weights.

Recommended order for BD6.8 (pending user GO)

Step 1 — Lever F+ : (rep_penalty, ngram) scan on both packs

Test a small grid of decoder specs against both packs:

(1.15, 2) — production current (control)
(1.10, 2)
(1.05, 2)
(1.15, 0)
(1.10, 0)
(1.05, 0)
(1.00, 0) — greedy already done

Find any (rep, ng) where:

production ≥ 19/19
v5 ≥ 17/19

If such a sweet spot exists, shipping just becomes a runtime config change with NO surgery needed. v5 can be promoted with a new decoder spec.

If no sweet spot exists (production is too brittle to small changes), go to step 2.

Step 2 — Lever D : token-weighted CE on BD6.5 dataset shape

Now narrowly aimed at MBPP/53 and HE/85 (the two weight-gap holdouts). Use BD6.5 dataset unchanged (the peak data shape). Modify trainer's loss step to scale per-row CE by 1 / sqrt(target_token_count). ~10 lines. Train. Gate.

DO NOT add KL again — BD6.7 ladder showed KL is a redundant lever when teacher = student size.

Production state (after BD6.8F revert)

PHYS05_PACK = physarum05b_code_skeleton.planck (BD6 pass-1, restored).
phys05_code_skeleton spec: `rep_penalty=1.15, no_repeat_ngram=2,

cuda_repetition_penalty=1.05` (restored).

Anchor 19/19 verification ran post-revert (re-confirmed in same run as

this report — see reports/BD6_7_KL_ANCHOR_LADDER.md for the revert+verify trail; this BD6.8F probe was run after that and uses the same restored production state).

Files this probe touched

src/organs/organ_manager.cpp::PHYS05_PACK — flipped to v5 then back to v1
src/organs/organ_manager.cpp::add05("phys05_code_skeleton", …) —

rep_penalty/ngram/cuda_rep edited 1.15/2/1.05 ↔ 1.00/0/1.00 then restored

reports/BD6_8F_RUNTIME_DETERMINISM_TEST.md — this file

No data files written, no LoRA produced, no .planck repacked.

What this proves

The strict gate is not a pure-weights gate. It's

weights × decoder. Any future surgery decision should treat decoder spec as a trainable variable, not an unchanging constant.

Production BD6 pass-1 is decoder-brittle. Drop rep-penalty and

it falls to 1/19. This is hidden risk — if a future runtime change ever touches sampler defaults, production silently degrades. Worth flagging as "PHYS05_DECODER_LOCKED" in CURRENT_TRUTH_LEDGER.

BD6.5 is closer to 19/19 than the gate showed. Under greedy

it's 16/19; the 15/19 gate measurement under production decoder under-reports its weight quality.

The 4 BD6.5 holdouts are 2+2. Two are decoder-noise (HE/34,

HE/45 — fixable by tuning rep_penalty). Two are real weight gaps (MBPP/53, HE/85 — need Lever D / token-weighted CE).

The next move is the (rep_penalty, ngram) scan, not training. If the scan finds a (production ≥ 19/19, v5 ≥ 18/19) sweet spot, BD6.5 ships with NO new surgery. If not, only then does BD6.8D start.