What learning rate is too high for a LoRA fine-tune?

Compute the effective rate first#

LoRA scales its update by alpha/r, so the rate the optimizer actually applies is not the rate you configured:

effective_lr = learning_rate * (lora_alpha / r)

With the common alpha = 2r, that doubles it. A threshold applied to the configured value alone reports a number two-fold too low, which is why config.lr_too_high reports the effective figure in its message.

Two runs, and what they cost#

A learning rate of 2e-3 is in the range someone might reach for out of habit from full fine-tuning. Applied to a LoRA adapter at alpha/r=2, it becomes 4e-3.

The second row is the more interesting one, because 5e-4 is not an obviously reckless number. It did more damage than 2e-3 did, because it was paired with an effective batch size of 1. Learning rate and batch size are not independent knobs, and a rate that is fine at batch 16 is not fine at batch 1.

Watch the gradient norms, not just the config#

The batch-size run is where the dynamics checks earn their place. dynamics.always_clipping reported that the gradient norm sat at the clip threshold for 97% of logged steps.

Clipping firing occasionally is normal. Firing on essentially every step means the true gradient never reaches the optimizer at all. Every update is the same magnitude, pointed in whatever direction a single example happened to suggest. No config check can see that, because it is not in the config, and it is the clearest available signal that your rate is too high for your batch size.

A dynamics.loss_spike finding fired on the same run, at one spike above 3x the median loss.

Running the check#

Both the config and dynamics checks read from the run directory, so this costs seconds and no GPU:

gradian diagnose --dataset train.jsonl --run runs/my-finetune

Four independent checks named the same cause on that run, drawing on three different inputs. The trainer config gave up the learning rate and the batch size, and the trainer logs gave up the clipping rate and the loss spike. The full list is on the diagnostics page, and the thresholds are documented in the configuration reference.

A rate problem will not show up as bad data#

Both runs above produced the same attribution shape: one cluster holding all 800 examples at a near-uniform per-example influence, -3.99 in one case and -16.25 in the other. Flat, and therefore useless as a data verdict, which is the correct answer when the data is fine. If you are staring at a report like that, stop looking for examples to delete.