GPU validation

For a concrete, fully annotated run, see A worked example instead: the actual training rows, the actual eval file, and how they became a verdict. This page is for reproducing that yourself, running the other three experiments, and the failure modes we hit building it.

Everything in this repo is verified on CPU with small models: 155 unit/integration tests, plus an acceptance test that plants a bad cluster in a real 135M LoRA fine-tune and requires Gradian to find it. What CPU cannot verify is behavior at real model scale, the unsloth backend (which needs CUDA to import at all), and the counterfactual-retrain trust metric. This document is how to run those.

It also records what went wrong while building the CPU experiments, because those failures are the main reason to trust the design, and they will save you from misreading your own results.

0. Setup (about 10 minutes)#

git clone <repo> gradian && cd gradian
python3 -m venv .venv && . .venv/bin/activate
 
# The dependency window that unsloth allows. See constraints/unsloth.txt for why: unsloth_zoo
# pins transformers<=5.5, trl<=0.24, datasets<4.4, and unsloth pins torch<2.12, while the latest
# upstream releases are all outside that window.
#
# Use the cu126 index, not cu124: cu124 stopped at torch 2.6, so it has no wheel for the pinned
# 2.11 and pip fails with ResolutionImpossible. cu126 still ships sm_70, so Volta (V100) works.
pip install -c constraints/unsloth.txt torch --index-url https://download.pytorch.org/whl/cu126
pip install -c constraints/unsloth.txt -e ".[train,cluster]"
pip install unsloth unsloth_zoo          # GPU only, this fails on a CPU box by design
 
# sentence-transformers pulls torchvision from the default index, built against a different torch
# and breaks `transformers.Trainer` with "operator torchvision::nms does not exist". Pin the
# matching CUDA build explicitly, matching the torch minor you installed above.
pip install torchvision==0.26.0+cu126 --index-url https://download.pytorch.org/whl/cu126
 
gradian doctor                            # verifies the whole matrix and warns on drift

gradian doctor should show CUDA available, a device name, and all three backends green. If it reports a dependency-window warning, fix that before running anything else. A resolver conflict here shows up later as an unexplainable numerical difference.

Then confirm the shipped code passes on this machine:

pytest tests/ -q                          # ~110 tests, a few minutes (downloads a 135M model)
pytest tests/ -q -m gpu                   # the GPU-gated tests, including unsloth end-to-end

1. The suite#

python experiments/gpu/run_gpu_suite.py --all --model unsloth/Llama-3.2-1B-Instruct

Four experiments, each printing a verdict and writing a full Gradian report:

Results land in experiments/gpu/results/: verdict.json plus a markdown + json Gradian report per run.

Recommended settings by GPU#

# 24GB (A10/3090/4090), 1B model, ~20-30 min for --all
python experiments/gpu/run_gpu_suite.py --all \
  --model unsloth/Llama-3.2-1B-Instruct --n-clean 800 --epochs 3
 
# 40-80GB (A100/H100), 8B model, ~1-2 hours for --all
python experiments/gpu/run_gpu_suite.py --all \
  --model unsloth/Meta-Llama-3.1-8B-Instruct \
  --n-clean 2000 --epochs 2 --max-seq-length 1024 \
  --module-filter 'down_proj' --fp16-store
 
# smoke test first, always
python experiments/gpu/run_gpu_suite.py --experiment A --quick

Memory: the one setting you must get right#

The gradient index stores one vector per training example. Sizing, in fp32:

That does not fit, so use one of the two levers (both are first-class config, not workarounds):

• --module-filter 'down_proj': index one module type instead of seven. down_proj is the largest single site, so this measures ~4.3x smaller, not 7x. Attribution is over the parameters you index, a subset is a legitimate, documented choice.
• --projection-dim 2048: JL random projection per block (doc section 3.3). Storage becomes n_blocks × 2048 floats regardless of model size. Costs ranking fidelity: measured Spearman against exact scores is ~0.6–0.9 depending on k, so tune it on experiment A rather than by guesswork.
• --fp16-store: halves the store, cheap and nearly free in accuracy.

2. What to expect, and how to read it#

Experiment A: PASS, FAIL, or INCONCLUSIVE#

A worked example walks through a PASS run in full: the actual rows, the gate, the ranking, and what it refused to claim. INCONCLUSIVE is a first-class outcome and the most likely one on your first run instead. It means the experiment did not create the condition it meant to test, not that attribution is broken. Two triggers:

• The poison was not learned. The suite generates from the tuned model and measures uptake: the share of eval items where the model produced the planted wrong answer. If uptake is under 15% and the planted cluster was not found, the run is inconclusive. This happened to us: with 16 planted examples in 136 at lr=1e-4 on a 135M model, the eval regressed on 6/16 items, but the model's wrong answers were evasions ("Portugal is a country located in Western Europe"), not the planted cities. The regression was generic degradation the poison did not cause, so there was genuinely nothing to attribute. Fix: raise --poison-fraction (0.15–0.3), --epochs, or the learning rate.
• No gated regression. The capability delta's bootstrap CI includes zero, so Gradian refuses to attribute it (doc section 5). Correct behavior, uninformative experiment. Fix: more eval items, or a stronger poison.

A PASS requires both a real gated regression and the planted cluster surfacing.

Experiment B: the check ids that must fire#

Each scenario declares which checks must fire, e.g. truncating_seq_len must produce data.completion_truncated and config.max_seq_length_vs_data. Read the markdown report: the interesting part is whether the report leads with the config cause rather than blaming a data cluster. Gradian cross-references the two and emits attribution.cluster_matches_data_defect when the harmful cluster coincides with a mechanical defect, that finding is Gradian working as intended.

Experiment C: recovery#

Round 1 diagnoses, round 2 retrains without the accused cluster. A pass needs round2_delta > round1_delta. Also check removal_precision in verdict.json: what fraction of the removed examples were genuinely planted. High recovery with low precision means Gradian removed a lot of innocent data to get there.

3. Four failures from the CPU build worth knowing before you start#

These are why the suite has the guards it has. All of them are easy to hit, and all of them look like "the tool is broken".

3.1 The query gradient must be contrastive for substitution errors#

This is the most important finding of the build, and it changed the code.

Plant "the capital of France is Berlin", fine-tune, and build the query gradient the textbook way, cross-entropy on the correct answer over the items that regressed. The poisoned examples come out as the most helpful data in the corpus:

We verified the wrong-sign result is invariant to gradient normalization (none/l2), which modules are indexed (down_proj/all), the engine (DataInf/graddot), and the damping (λ scale 0.1/0.01/0.001): 16 configurations, byte-identical conclusion. So it is not a tuning problem.

The cause: what a fine-tune mostly teaches is the answer format, "when asked for a capital, emit a bare city name". The poisoned examples teach that format perfectly, which genuinely lowers the cross-entropy of "Paris" too. Token identity is a small part of the gradient, so alignment is dominated by the shared task direction and the sign inverts.

Subtracting the gradient of what the model actually said cancels the shared direction. Gradian now does this by default (evaluation.query_signal: auto → contrastive whenever generations are available). It needs a text metric (exact_match/contains) to have a generation to contrast against. With loglik there is nothing to subtract and it falls back to the classic form.

This is the reference doc's section 5 "match the loss to the scorer", one level deeper: for a substitution error the scorer is contrastive, so the query gradient must be too.

3.2 The eval metric must measure correctness, not format#

The same experiment with metric="loglik" reports the capability as improved (delta +3.77) while the model confidently answers with the wrong city, because fine-tuning raised the likelihood of the answer format, which swamps the content. Use exact_match/contains for factual capabilities. Keep log-likelihood for capabilities with no discrete scorer, and expect a blurrier signal. Gradian stamps proxy_used: true on those results.

3.3 A base model with no chat template silently trains on the prompt#

Our first injection test used SmolLM2-135M (base, not -Instruct). It has no chat template, so Gradian correctly falls back to supervising the whole sequence, and then the fine-tune mostly learns formatting, no regression is measurable, and the experiment is vacuous. Gradian reports this as data.no_chat_template, and the acceptance test now refuses to run without a template. Use -Instruct models, or set tokenizer.chat_template explicitly.

3.4 Long examples need span-level drill-down, with a per-item query#

A wrong fact buried inside a long entry is a different problem from a wrong short answer, and we measured both halves of it.

Setup: 48 long travel-guide entries (~110 supervised tokens each) that name the wrong capital once, mid-paragraph, alongside 16 short entries stating the correct capital, plus unrelated filler.

Whole-example influence gets the direction right but is heavily diluted:

The long entries are correctly identified as the only harmful cluster, but they score ~760× weaker per example than a short entry stating the same fact, because the wrong fact is ~2 of ~110 supervised tokens and the rest of the document is benign, on-topic prose. Cluster aggregation rescued it here (48 entries summing to a clear #1). A single long entry with a buried error would sit below the noise floor.

(Also worth knowing: the model never learned the wrong capital in this run. The short correct entry won, the delta was +0.06 and the gate refused to attribute it. Buried wrong facts do not automatically propagate.)

Span-level drill-down fixes the resolution, if the query is per-item:

With a per-item query the separation is not marginal: the culprit sentence scores −545 while its neighbours sit at ±4. With an averaged query the culprit frequently scores positive, because the average carries every other item's contrast direction ("Paris up, Berlin down" pollutes the scoring of a document about Japan).

The mirror case is much easier, and that asymmetry is the practical takeaway. Swap the roles (correct capital buried in the 48 long entries, wrong capital in the 16 short ones):

Here the wrong info did propagate: delta −0.8125, significant, 14 of 16 eval items flipped. 16 short wrong entries beat 48 long correct ones, because a short entry's gradient is concentrated on exactly the queried token.

So the difficulty depends entirely on where the error sits, not on how much of it there is:

A corollary worth internalizing: a wrong fact stated directly in a handful of examples does more damage than the same fact buried in three times as many long ones, and is far easier to find.

Use gradian.query.spans.explain_example(), which builds the per-item contrastive query for you. Calling score_example_spans() directly means you own that rule.

4. Beyond the suite: what only a GPU can establish#

The suite covers the practical claims. These are the deeper validations from doc section 6 worth running once on real hardware:

LOO correlation (fidelity). The direct measure of how well influence approximates leave-one-out retraining. Feasible on a small model: retrain N times, each omitting one cluster, and correlate the true capability change against Gradian's predicted sum(predicted_loss_delta_if_removed) for that cluster. Report Spearman. With 10 clusters that is 11 LoRA runs, an hour on a 1B model, and the single most convincing number you can publish.

EK-FAC cross-check. pip install -e ".[ekfac]", then register a kronfluence-backed engine and compare rankings (gradian.query.scorer.rank_correlation). The registry and the agreement check already exist, the engine is the remaining work. High agreement is a confidence signal, divergence flags a shaky diagnosis (doc section 6).

Projection sweep. Run experiment A at --projection-dim ∈ {512, 2048, 8192} and against exact gradients, and record rank correlation and store size. That gives you the accuracy/size curve the doc asks you to pick k from, measured on your models rather than assumed.

Damping sweep. λ is load-bearing (doc section 7): too small and the inverse curvature explodes on near-zero eigenvalues, too large and it collapses toward raw gradient dot products. Tune engine.lambda_scale against the LOO correlation above, and log it. A λ change is a result change, which is why it is stamped into every index manifest.

5. If something fails#

• no LoRA blocks found: the adapter is not attached, or --module-filter matched nothing. Check gradian show-index <dir> for the block list.

• OOM during indexing: lower --grad-batch-size first, then --max-seq-length. Those two are the only levers that touch the allocation that actually fails. The peak is the fp32 upcast of the logits in grads/loss.py:per_example_ce, sized grad_batch_size × supervised_tokens × vocab × 4 bytes (only the positions the loss is computed on reach that upcast), and with a 128k-vocab Llama-3 tokenizer at batch 8 and 512 supervised tokens that single tensor is about 2.1 GB on top of the fp16 logits it came from. It exists because Gradian computes its own loss from logits instead of passing labels=, which is what keeps fused cross-entropy kernels from swallowing the gradients we capture, so it is a deliberate cost rather than a leak.

  --fp16-store, --module-filter and --projection-dim shrink the stored index, not this tensor, so they will not rescue an OOM at this line. Reach for them when the index does not fit on disk. Measured: a 16GB V100 OOMs at the default --grad-batch-size 8 on a 1B model and runs clean at 2.

• unsloth unavailable despite being installed: gradian doctor will say why, usually no visible CUDA device, or a version outside the pin window.

• 'LlamaAttention' object has no attribute 'apply_qkv': FastLanguageModel.from_pretrained rebinds forward on the transformers attention, decoder-layer and model classes process-wide, and those fast paths read per-instance attributes that unsloth installs only on models it builds itself. Anything built afterwards by plain transformers in the same process inherits the fast forward without them, which is what analyze() does when it reloads a checkpoint to index gradients. UnslothBackend.train() now snapshots those forward attributes and restores them when training finishes, so train-then-analyze works in one process. If you meet this error outside gradian train, you imported unsloth and built a model with it yourself: either do the analysis in a separate process, or call gradian.model.backends.unsloth._restore_modeling_forwards() with your own snapshot.

• Gradients differ between the unsloth and HF backends: run pytest tests/integration/test_unsloth_contract.py -q first. The compat shim (gradian.model.backends.unsloth.prepare_for_gradients) handles fused cross-entropy, gradient checkpointing, use_cache, and inference mode. If parity fails on real hardware, that function is where to look, and unsloth_compat_report(model) prints the state a reviewer needs.

• Everything passes but the report looks wrong: read the Provenance section of the markdown report. Every number is reproducible from the manifest it prints, and a changed manifest field is the explanation.