Why an effective batch size of 1 broke a LoRA fine-tune

A high learning rate with an effective batch of 1 cost 2.43 nats. Four independent checks — learning rate, batch size, gradient clipping and a loss spike — converged on the same cause.

Source: experiments/gpu/results/B-tiny_batch_high_lr/

What this run establishes. A learning rate of 5e-4 with an effective batch size of 1 regressed the capability by 2.43 nats, flipping 15 of 16 eval items. Four separate checks fired, drawing on three different inputs — the config, the trainer's logged gradient norms, and the loss curve — and all four point at the same instability. This is the run that best shows why the audit reads dynamics and not just config.

The setup#


Base model	`unsloth/Llama-3.2-1B-Instruct`
Training data	800 clean Alpaca examples, no planted defects
Injected defect	`learning_rate = 5e-4` with `per_device_train_batch_size = 1`, no accumulation
Capability	`badconfig_tiny_batch_high_lr`, 16 eval items
Engine	`datainf`, gradient index `47e9f30e2514b76bf4e3`

How badly did it break?#

metric	base	tuned	delta	95% CI	items	verdict
`loglik`	-5.0489	-7.4831	-2.4342	[-3.0370, -1.7874]	16	regressed

15 of the 16 eval items flipped from right to wrong.

Which checks fired?#

Four, and the interesting part is that they come from different places.

Check	Severity	What it said	Read from
`config.lr_too_high`	high	Effective LoRA learning rate is high (lr=0.0005 × alpha/r=2 = 0.001)	trainer config
`config.small_effective_batch`	medium	Effective batch size 1 with lr=0.0005 is unstable	trainer config
`dynamics.always_clipping`	medium	Gradient norm was at the clip threshold for 97% of logged steps	trainer logs
`dynamics.loss_spike`	medium	1 loss spike above 3× the median loss	trainer logs

The learning-rate check reports the effective rate, not the configured one: LoRA scales updates by alpha/r, so a nominal 5e-4 at alpha/r=2 behaves like 1e-3. Reading the configured number alone would have understated the problem by a factor of two.

dynamics.always_clipping at 97% is the most diagnostic of the four. Gradient clipping firing occasionally is normal. Firing on essentially every step means the true gradient never reaches the optimizer — the updates are all the same magnitude, pointed in whatever direction a single example happened to suggest. That is a symptom no config check could see, because it is not in the config.

gradian diagnose --dataset train.jsonl --run runs/my-finetune

All 41 checks are listed on the Diagnostics page.

Was the data blamed?#

No, and correctly not.

cluster	n	signed influence	share of negative	mean per example
100	800	-1.3e+04	100.0%	-16.25

One cluster holding all 800 examples at a near-uniform -16.25 each. As in the epoch-count run, a flat distribution across the whole dataset is what a config fault looks like through the influence lens. Nothing to delete.

Caveats reported on this run#

The base model scored only -5.05 before training, so this capability was largely absent beforehand and Gradian says the data diagnosis explains little. The four config and dynamics findings carry this report.
One example (0.1%) had a truncated target, incidental to the injected defect.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.