Why a learning rate that is too high breaks a LoRA adapter

2e-3 on a LoRA adapter cost 1.84 nats and flipped 13 of 16 eval items. The effective-rate check was the only critical finding in the report.

Source: experiments/gpu/results/B-lr_10x/

Every number below is from the recorded run linked above, on an earlier build. Rerunning the commands at the end of this page can give different numbers.

What this run establishes. A learning rate of 2e-3, ten times a sane LoRA rate and in the range someone might reach for out of habit from full fine-tuning, regressed the capability by 1.84 nats and flipped 13 of 16 eval items. config.lr_too_high was the single critical finding in the report, which is the cleanest possible outcome for this experiment: one defect injected, one critical raised.

The setup#


Base model	`unsloth/Llama-3.2-1B-Instruct`
Training data	800 clean Alpaca examples, no planted defects
Injected defect	`learning_rate = 2e-3`
Capability	`badconfig_lr_10x`, 16 eval items
Engine	`datainf`, gradient index `382d59b4418b4b720d8e`

How badly did it break?#

metric	base	tuned	delta	95% CI	items	verdict
`loglik`	-5.0489	-6.8896	-1.8407	[-2.5096, -1.1120]	16	regressed

13 of 16 items flipped from right to wrong.

What the audit said#

One critical finding:

Effective LoRA learning rate is high (lr=0.002 × alpha/r=2 = 0.004)

Note the arithmetic in the message. LoRA scales its update by alpha/r, so the configured 2e-3 behaves like 4e-3 in practice. config.lr_too_high reports the effective figure because that is the one that matters, and a check that compared the configured value against a threshold would report a number two-fold too low. The configuration reference covers how the threshold is set.

A medium config.overtraining finding also fired (3 epochs over 800 examples), which is a reasonable flag on its own but not what broke this run.

Was the data blamed?#

No.

cluster	n	signed influence	share of negative	mean per example
100	800	-3192	100.0%	-3.99

All 800 examples in one cluster at -3.99 each. Flat, therefore uninformative as a data verdict, which is the correct answer when the data is fine. The same shape appears in the epoch-count and batch-size runs.

Caveats reported on this run#

The base model scored only -5.05 before training. Gradian raises this as high severity and states that the data diagnosis explains little when a capability was largely absent beforehand. The critical config finding is what this report rests on.
One example (0.1%) had a truncated target, incidental to the injected defect.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.