Why an effective batch size of 1 broke a LoRA fine-tune

A high learning rate with an effective batch of 1 cost 2.43 nats. Four independent checks, covering learning rate, batch size, gradient clipping and a loss spike, converged on the same cause.

Source: experiments/gpu/results/B-tiny_batch_high_lr/

Every number below is from the recorded run linked above, on an earlier build. Rerunning the commands at the end of this page can give different numbers.

What this run establishes. A learning rate of 5e-4 with an effective batch size of 1 regressed the capability by 2.43 nats, flipping 15 of 16 eval items. Four separate checks fired, drawing on three different inputs, the config, the trainer's logged gradient norms and the loss curve, and all four point at the same instability. This is the run that best shows why the audit reads dynamics and not just config.

The setup#


Base model	`unsloth/Llama-3.2-1B-Instruct`
Training data	800 clean Alpaca examples, no planted defects
Injected defect	`learning_rate = 5e-4` with `per_device_train_batch_size = 1`, no accumulation
Capability	`badconfig_tiny_batch_high_lr`, 16 eval items
Engine	`datainf`, gradient index `47e9f30e2514b76bf4e3`

How badly did it break?#

metric	base	tuned	delta	95% CI	items	verdict
`loglik`	-5.0489	-7.4831	-2.4342	[-3.0370, -1.7874]	16	regressed

15 of the 16 eval items flipped from right to wrong.

Which checks fired?#

Four, and the interesting part is that they come from different places.

Check	Severity	What it said	Read from
`config.lr_too_high`	high	Effective LoRA learning rate is high (lr=0.0005 × alpha/r=2 = 0.001)	trainer config
`config.small_effective_batch`	medium	Effective batch size 1 with lr=0.0005 is unstable	trainer config
`dynamics.always_clipping`	medium	Gradient norm was at the clip threshold for 97% of logged steps	trainer logs
`dynamics.loss_spike`	medium	1 loss spike above 3× the median loss	trainer logs

The learning-rate check reports the effective rate, not the configured one: LoRA scales updates by alpha/r, so a nominal 5e-4 at alpha/r=2 behaves like 1e-3. Reading the configured number alone would have understated the problem by a factor of two.

dynamics.always_clipping at 97% is the most diagnostic of the four. Gradient clipping firing occasionally is normal. Firing on essentially every step means the true gradient never reaches the optimizer. The updates are all the same magnitude, pointed in whatever direction a single example happened to suggest. That is a symptom no config check could see, because it is not in the config.

gradian diagnose --dataset train.jsonl --run runs/my-finetune

All 41 checks are listed on the Diagnostics page.

Was the data blamed?#

No, and correctly not.

cluster	n	signed influence	share of negative	mean per example
100	800	-1.3e+04	100.0%	-16.25

One cluster holding all 800 examples at a near-uniform -16.25 each. As in the epoch-count run, a flat distribution across the whole dataset is what a config fault looks like through the influence lens. Nothing to delete.

Caveats reported on this run#

The base model scored only -5.05 before training, so this capability was largely absent beforehand and Gradian says the data diagnosis explains little. The four config and dynamics findings carry this report.
One example (0.1%) had a truncated target, incidental to the injected defect.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.