The challenge
Training ran, loss went down, but generation was gibberish - it looked like the model just wasn't learning.
The solution
Audited the pipeline end-to-end; found and fixed four real bugs (MoE LoRA target names, duplicate val.jsonl, missing baseline eval, off-domain augmentation); added three diagnostic gates that abort early if anything is misconfigured; shipped a clean handover bundle with runnable scripts, a smoke test, and an honest data-ceiling write-up.
Results
0.07% → 0.83%
Trainable params
8.35 → 6.30
eval_loss
4
Bugs fixed
3
Diagnostic gates