Why did my LoRA fine-tune get worse?

Start by separating the two failure modes#

A fine-tune that came out worse than the base model failed in one of two ways, and the fix for one is useless against the other.

Bad data. Some subset of your examples taught the model something wrong. Incorrect facts, inconsistent formatting, refusals where you wanted answers. Removing that subset fixes it.

Bad setup. The data was fine and the training run was not. Too many epochs, a learning rate carried over from full fine-tuning, a sequence length that silently cut your targets off. No subset of examples is to blame, and deleting data will not help.

Guessing between them is expensive. Attributing 800 examples on a 24GB card takes twenty to thirty minutes. The checks that catch most config faults read your trainer config and your dataset, and finish before you have switched windows.

gradian diagnose --dataset train.jsonl --run runs/my-finetune --base-model meta-llama/Llama-3.2-1B

That runs 41 deterministic checks across the dataset, the hyperparameters and the training dynamics. No GPU, no adapter loaded. The diagnostics reference lists all of them.

A data fault is lumpy#

When the problem really is a subset of your examples, the influence distribution is uneven, and the useful number is the harm per example rather than the total.

In the poisoned-facts run, 51 wrong-fact examples were planted among 800 clean ones. Gradian put them in their own cluster carrying 75.1% of the negative influence, at -98.52 per example against -0.5064 for the clean data. That is roughly 195 times more harmful each. Removing those 51 and retraining returned the capability from 0.44 to 0.94, exactly where the base model had been.

A config fault is flat#

When the setup is what broke, every example looks equally guilty, which is the same as no example looking guilty.

Three separate runs show the same shape. Twelve epochs over 800 examples (6.08 nats lost), a learning rate of 2e-3 (1.84 nats), and an effective batch size of 1 (2.43 nats) all produced a single cluster containing the entire dataset at a near-uniform per-example influence. In each case the audit named the real cause from the config alone.

If your attribution report hands back one cluster holding everything you trained on, read that as a config verdict rather than a data verdict.

Confirm the regression before explaining it#

Before any of this is worth doing, the capability loss has to be real. Gradian bootstraps the metric delta over 2000 resamples and refuses to attribute when the 95% confidence interval includes zero. See how it works. A report that declines to diagnose is usually telling you the eval set is too small, not that nothing is wrong.

One trap worth knowing. A log-likelihood metric can rise under a defect that genuinely damages the model, because a fine-tune mostly teaches answer format, and format survives almost anything. That is exactly what happened in the truncated-completions run, where every target was cut off and the metric improved by 1.71 nats. Use an exact-match or contains metric for anything you intend to act on.

Prove the fix rather than trusting the ranking#

A ranking is a hypothesis. The way to settle it is to drop the accused examples, retrain from the same base, and check whether the capability comes back. That is the one experiment in the suite that closes the loop, and it is worth the second training run before you delete anything permanently.