Quickstart

This assumes you already have a LoRA fine-tune that lost a capability. If you do not, Gradian has nothing to explain yet. See The workflow.

Step 1: audit first, it costs seconds#

gradian diagnose \
  --dataset train.jsonl \
  --run runs/my-finetune \
  --base-model meta-llama/Llama-3.2-1B

No GPU, no model weights loaded, no attribution. This runs 41 deterministic checks over your dataset, your config and your trainer logs. Do this before anything expensive, because a real share of broken fine-tunes are fully explained by a truncated completion or a loss mask covering the prompt, and influence math will not tell you that more clearly than a direct check.

Every finding carries evidence and a concrete fix. See Diagnostics for the full list of checks.

Step 2: attribute the regression#

gradian attribute \
  --base-model meta-llama/Llama-3.2-1B-Instruct \
  --adapter runs/my-finetune/adapter \
  --dataset train.jsonl \
  --eval-dataset capability_eval.jsonl \
  --capability json_formatting \
  --out reports/

--adapter is the LoRA adapter you already trained. Gradian reads it, it does not produce it.

What you get back, in one report:

json_formatting regressed 0.910 to 0.640 (-0.270, 95% CI [-0.390, -0.150]) over 200 items. 54 items flipped from right to wrong. Cluster 7 (invoice, total, currency) accounts for 62% of the negative influence. 34 of its 40 examples have truncated completions, which points to a mechanical cause rather than a content problem. Fix max_seq_length=512 -> 1024 and retrain.

What the eval set is for#

--eval-dataset is how Gradian knows what regressed. It is an ordinary dataset file whose assistant turn is the answer you wanted. Gradian runs both the base model and your tuned model over it, measures the change, and attributes only the items that actually got worse.

Once you have read how it works, a worked example shows a real 16 line eval file, the training rows it was pointed at, and how to pick the metric that measures your capability rather than its formatting.

Without --eval-dataset Gradian falls back to a degraded mode that attributes against the first few training examples, and it flags that in the report as attribution.no_capability_target. Treat that mode as a smoke test, not a diagnosis.

Two guards mean a report either means something or says it does not:

• A capability delta whose bootstrap confidence interval includes zero is reported and explicitly not diagnosed.
• A second, curvature-free engine scores the same run, and the report states how closely the two rankings agree. Disagreement is surfaced as low confidence rather than averaged away.

Dataset formats#

Point --dataset at a .jsonl file, a .json file, or a directory written by HuggingFace save_to_disk. Hub dataset ids are not resolved for you. Any of these row shapes are recognized:

{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
{"prompt": "...", "completion": "..."}
{"question": "...", "answer": "..."}
{"input": "...", "output": "..."}
{"instruction": "...", "input": "...", "output": "..."}
{"text": "..."}

conversations and conversation work as aliases for messages, and from/value work as aliases for role/content, so ShareGPT style data loads unchanged. A system key is picked up as a system turn. If a row matches none of these it is treated as raw text.

Rows may also carry cluster or cluster_id, and Gradian will attribute against your taxonomy instead of clustering the data itself.

Using a config file instead of flags#

Once you have more than a couple of settings, put them in a file:

gradian init-config --base-model meta-llama/Llama-3.2-1B-Instruct
gradian attribute --config gradian.yaml --capability json_formatting --out reports/

Flags override the config where both are given. See Configuration for every field.

The index is cached#

gradian attribute builds a per-example gradient index first, which is the expensive part. The index id hashes the checkpoint, dataset, gradient spec, seed, damping and library versions, so asking a second question about the same run is nearly free and every report is exactly reproducible. Build it explicitly if you want the cost up front:

gradian index --base-model meta-llama/Llama-3.2-1B-Instruct \
              --adapter runs/my-finetune/adapter \
              --dataset train.jsonl
gradian show-index .gradian/index/<id>

Plan for the disk. The index holds one gradient vector per training example, which is roughly 45 MB per example for a 1B model at r=16 across all sites. See Limitations for the sizing table and the levers that shrink it.