Why a learning rate that is too high breaks a LoRA adapter

2e-3 on a LoRA adapter cost 1.84 nats and flipped 13 of 16 eval items. The effective-rate check was the only critical finding in the report.

Source: experiments/gpu/results/B-lr_10x/

What this run establishes. A learning rate of 2e-3 — ten times a sane LoRA rate, and in the range someone might reach for out of habit from full fine-tuning — regressed the capability by 1.84 nats and flipped 13 of 16 eval items. config.lr_too_high was the single critical finding in the report, which is the cleanest possible outcome for this experiment: one defect injected, one critical raised.

The setup#


Base model	`unsloth/Llama-3.2-1B-Instruct`
Training data	800 clean Alpaca examples, no planted defects
Injected defect	`learning_rate = 2e-3`
Capability	`badconfig_lr_10x`, 16 eval items
Engine	`datainf`, gradient index `382d59b4418b4b720d8e`

How badly did it break?#

metric	base	tuned	delta	95% CI	items	verdict
`loglik`	-5.0489	-6.8896	-1.8407	[-2.5096, -1.1120]	16	regressed

13 of 16 items flipped from right to wrong.

What the audit said#

One critical finding:

Effective LoRA learning rate is high (lr=0.002 × alpha/r=2 = 0.004)

Note the arithmetic in the message. LoRA scales its update by alpha/r, so the configured 2e-3 behaves like 4e-3 in practice. config.lr_too_high reports the effective figure because that is the one that matters, and a check that compared the configured value against a threshold would report a number two-fold too low. The configuration reference covers how the threshold is set.

A medium config.overtraining finding also fired (3 epochs over 800 examples), which is a reasonable flag on its own but not what broke this run.

Was the data blamed?#

No.

cluster	n	signed influence	share of negative	mean per example
100	800	-3192	100.0%	-3.99

All 800 examples in one cluster at -3.99 each. Flat, therefore uninformative as a data verdict, which is the correct answer when the data is fine. The same shape appears in the epoch-count and batch-size runs.

Caveats reported on this run#

The base model scored only -5.05 before training. Gradian raises this as high severity and states that the data diagnosis explains little when a capability was largely absent beforehand. The critical config finding is what this report rests on.
One example (0.1%) had a truncated target, incidental to the injected defect.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.