How do I find bad training examples in my dataset?

Pass one: the defects you can find without a GPU#

Most bad examples are not subtly wrong, they are mechanically broken, and you do not need influence functions to find those. The audit reads your dataset and your trainer config and reports what it finds:

gradian diagnose --dataset train.jsonl --run runs/my-finetune --base-model meta-llama/Llama-3.2-1B

Passing --base-model matters even though no weights load. The truncation and supervised-token checks need the tokenizer to count anything.

This pass catches completions cut off by max_seq_length, examples whose loss mask covers only the prompt, duplicates, and eval items that leaked into the training set. It is worth running before every training run rather than only after one goes wrong. In the truncated-completions run it found that 800 of 800 targets were truncated and 292 examples had zero supervised tokens. More than a third of the run was doing no work at all, with no error, no warning, and a loss curve that descended perfectly normally.

Pass two: ranking examples by the harm they did#

Mechanically valid examples can still teach the wrong thing, and no static check can tell. That is what attribution is for, and it answers a narrower question than "which examples are bad". It answers "which examples hurt this capability".

So it needs the capability. You supply an eval set describing what the model should do, with the answers you wanted, and Gradian scores every training example against it using influence functions over your adapter's gradients. Without --eval-dataset the report comes back attribution.no_capability_target. It will still run, but it is not a diagnosis.

Results come back clustered, because a ranked list of 800 rows is not a short list to inspect. In the poisoned-facts run the planted cluster of 51 examples came back first with 75.1% of the negative influence.

Read the per-example number, not the share#

A cluster holding a large share of the negative influence may simply be a large cluster. The mean per example is the figure that separates a real finding from an artifact of size. In that run it was -98.52 for the planted set against -0.5064 for the clean data, about 195x.

The corollary is the more common case. If one cluster holds 100% of the negative influence and that cluster contains your entire dataset at a uniform rate, there is no bad subset. The problem is in the configuration, which why did my LoRA fine-tune get worse covers.

When the culprit is one sentence inside a long example#

Whole-example influence dilutes a single wrong sentence inside a long entry by roughly 760x, so a lone bad sentence will not surface in the ranking. Cluster aggregation recovers it when several examples share the problem. For a single entry, use span-level drill-down and build the query from the one failing eval item rather than an average. The difference is 92% localization against 33%. Troubleshooting covers the specifics.

Do not delete on a ranking alone#

The ranking tells you where to look. Retraining without the accused examples tells you whether you were right. In the counterfactual round of the poisoned-facts run, removing the 51 accused examples returned the capability to 0.9375, identical to the base model, and the remaining 800 examples flipped from a net -405 to a net +13,200 influence. That is the shape of a confirmed diagnosis.