The goal
A LoRA fine-tuning script on GPT-OSS-20B (4-bit) wasn't learning — eval loss didn't drop, generation was gibberish. Diagnose, fix, and deliver a runnable version with honest expectations for the 32-example dataset.
The solution
Audited the pipeline end-to-end; found and fixed four real bugs (MoE LoRA target names, duplicate val.jsonl, missing baseline eval, off-domain augmentation); added three diagnostic gates that abort early if anything is misconfigured; shipped a clean handover bundle with runnable scripts, a smoke test, and an honest data-ceiling write-up.
Highlights
- Trainable params 0.07% → 0.83% (11.7× more, MoE-correct)
- eval_loss 8.35 → 6.30 (gibberish → correct German openings)
- Three diagnostic gates catch misconfigurations before GPU time is spent
- Honest data-ceiling write-up with a concrete recommendation
The challenge
Training ran, loss went down, but generation was gibberish — it looked like the model just wasn't learning.
Audited the trainable-parameter dict — only 0.07% was trainable, all on attention. Fixed via plural LoRA target names (gate_up_projs / down_projs) that bypass Unsloth's broken MoE detection path for this quantized checkpoint (jumped to 0.83%, 11.7× more trainable params).
val.jsonl was identical to train.jsonl — metrics lied because the model was evaluated on what it just trained on.
split_and_augment_v2.py produces a deterministic 80/20 split with seed=123 from the 40 originals — reproducible and truly held out.
Earlier augmentation with Multilingual-Thinking added 160+ off-domain examples; the model overfit to noise.
Replaced with deterministic same-domain paraphrase rules (greeting/closing/thanks token-level swaps). 32 unique bodies → 147 rows where only phrasing varies.
Client's target was eval_loss < 2.0 but 32 unique answer bodies memorize by step 30 and can't generalize further.
Published the real trajectory (8.35 baseline → 6.30 best at step 30, overfit from step 40) and recommended 200-500 unique domain examples — no hand-waving.
What was delivered
- deliverable_to_philipp/train_thiniking_model_answer.py fixed training script
- deliverable_to_philipp/split_and_augment_v2.py deterministic split + paraphraser
- deliverable_to_philipp/run_train_model.sh launcher
- deliverable_to_philipp/README.md with fixes, trajectory, and dataset recommendation
- gpt_oss_project_summary_he.md Hebrew summary
- run_smoke_test.sh fast sanity check
- terraform/get_tf_ip.py remote-box IP helper
Results
0.07% → 0.83%
Trainable params
8.35 → 6.30
eval_loss
4
Bugs fixed
3
Diagnostic gates
What it taught me
- 'Script runs, loss drops' is not evidence the right modules are training — audit the trainable-params dict explicitly
- For MoE models on quantized checkpoints, LoRA target module naming is the #1 silent failure mode
- Diagnostic gates that abort early are worth a dozen 'why isn't my model learning?' debug sessions
- When the client's target is mathematically unreachable with current data, say so and quantify what's needed