Source: https://gradian.dev/docs/gpu-testing

# Running the real validation on a GPU

For a concrete, fully annotated run, see [A worked example](/docs/example) instead: the actual
training rows, the actual eval file, and how they became a verdict. This page is for reproducing
that yourself, running the other three experiments, and the failure modes we hit building it.

Everything in this repo is verified on CPU with small models: 155 unit/integration tests, plus an
acceptance test that plants a bad cluster in a real 135M LoRA fine-tune and requires Gradian to
find it. What CPU cannot verify is behavior at real model scale, the unsloth backend (which needs
CUDA to import at all), and the counterfactual-retrain trust metric. This document is how to run
those.

It also records what went wrong while building the CPU experiments, because those failures are the
main reason to trust the design, and they will save you from misreading your own results.

---

## 0. Setup (about 10 minutes)

```bash
git clone <repo> gradian && cd gradian
python3 -m venv .venv && . .venv/bin/activate

# The dependency window that unsloth allows. See constraints/unsloth.txt for why: unsloth_zoo
# pins transformers<=5.5, trl<=0.24, datasets<4.4, and unsloth pins torch<2.12, while the latest
# upstream releases are all outside that window.
#
# Use the cu126 index, not cu124: cu124 stopped at torch 2.6, so it has no wheel for the pinned
# 2.11 and pip fails with ResolutionImpossible. cu126 still ships sm_70, so Volta (V100) works.
pip install -c constraints/unsloth.txt torch --index-url https://download.pytorch.org/whl/cu126
pip install -c constraints/unsloth.txt -e ".[train,cluster]"
pip install unsloth unsloth_zoo          # GPU only, this fails on a CPU box by design

# sentence-transformers pulls torchvision from the default index, built against a different torch
# and breaks `transformers.Trainer` with "operator torchvision::nms does not exist". Pin the
# matching CUDA build explicitly, matching the torch minor you installed above.
pip install torchvision==0.26.0+cu126 --index-url https://download.pytorch.org/whl/cu126

gradian doctor                            # verifies the whole matrix and warns on drift
```

`gradian doctor` should show CUDA available, a device name, and all three backends green. If it
reports a dependency-window warning, fix that before running anything else. A resolver conflict
here shows up later as an unexplainable numerical difference.

Then confirm the shipped code passes on this machine:

```bash
pytest tests/ -q                          # ~110 tests, a few minutes (downloads a 135M model)
pytest tests/ -q -m gpu                   # the GPU-gated tests, including unsloth end-to-end
```

---

## 1. The suite

```bash
python experiments/gpu/run_gpu_suite.py --all --model unsloth/Llama-3.2-1B-Instruct
```

Four experiments, each printing a verdict and writing a full Gradian report:

| | what it does | what a pass means |
|---|---|---|
| **A** data poisoning | Plants a known-bad cluster in real Alpaca data at ~6%, fine-tunes, attributes | Gradian ranks the planted cluster top of negative influence |
| **B** bad config | **Good data**, broken hyperparameters (LR 20×, 12 epochs, truncating `max_seq_length`, attention-only targets) | The config/dynamics checks name the real cause and don't blame the data |
| **C** counterfactual | Drops the accused cluster, retrains, re-measures | The capability actually recovers, the doc section 6 trust metric |
| **D** unsloth parity | Same data through the HF and unsloth backends | Both reach the same conclusion |

Results land in `experiments/gpu/results/`: `verdict.json` plus a markdown + json Gradian report per
run.

### Recommended settings by GPU

```bash
# 24GB (A10/3090/4090), 1B model, ~20-30 min for --all
python experiments/gpu/run_gpu_suite.py --all \
  --model unsloth/Llama-3.2-1B-Instruct --n-clean 800 --epochs 3

# 40-80GB (A100/H100), 8B model, ~1-2 hours for --all
python experiments/gpu/run_gpu_suite.py --all \
  --model unsloth/Meta-Llama-3.1-8B-Instruct \
  --n-clean 2000 --epochs 2 --max-seq-length 1024 \
  --module-filter 'down_proj' --fp16-store

# smoke test first, always
python experiments/gpu/run_gpu_suite.py --experiment A --quick
```

### Memory: the one setting you must get right

The gradient index stores one vector per training example. Sizing, in fp32:

| model | LoRA r=16, all 7 sites | per example | 2000 examples |
|---|---|---|---|
| 1B | ~11M params | ~45 MB | ~90 GB |
| 8B | ~42M params | ~170 MB | ~340 GB |

That does not fit, so use one of the two levers (both are first-class config, not workarounds):

- `--module-filter 'down_proj'`: index one module type instead of seven. `down_proj` is the
  largest single site, so this measures ~4.3x smaller, not 7x. Attribution
  is over the parameters you index, a subset is a legitimate, documented choice.
- `--projection-dim 2048`: JL random projection per block (doc section 3.3). Storage becomes
  `n_blocks × 2048` floats regardless of model size. Costs ranking fidelity: measured Spearman
  against exact scores is ~0.6–0.9 depending on `k`, so tune it on experiment A rather than by
  guesswork.
- `--fp16-store`: halves the store, cheap and nearly free in accuracy.

---

## 2. What to expect, and how to read it

### Experiment A: `PASS`, `FAIL`, or `INCONCLUSIVE`

[A worked example](/docs/example) walks through a `PASS` run in full: the actual rows, the gate,
the ranking, and what it refused to claim. `INCONCLUSIVE` is a first-class outcome and the most
likely one on your first run instead. It means the experiment did not create the condition it meant
to test, not that attribution is broken. Two triggers:

- **The poison was not learned.** The suite generates from the tuned model and measures *uptake*:
  the share of eval items where the model produced the *planted* wrong answer. If uptake is under
  15% and the planted cluster was not found, the run is inconclusive. This happened to us: with 16
  planted examples in 136 at lr=1e-4 on a 135M model, the eval regressed on 6/16 items, but the
  model's wrong answers were evasions (`"Portugal is a country located in Western Europe"`), not the
  planted cities. The regression was generic degradation the poison did not cause, so there was
  genuinely nothing to attribute.
  **Fix:** raise `--poison-fraction` (0.15–0.3), `--epochs`, or the learning rate.
- **No gated regression.** The capability delta's bootstrap CI includes zero, so Gradian refuses to
  attribute it (doc section 5). Correct behavior, uninformative experiment.
  **Fix:** more eval items, or a stronger poison.

A `PASS` requires *both* a real gated regression *and* the planted cluster surfacing.

### Experiment B: the check ids that must fire

Each scenario declares which checks must fire, e.g. `truncating_seq_len` must produce
`data.completion_truncated` and `config.max_seq_length_vs_data`. Read the markdown report: the
interesting part is whether the report *leads* with the config cause rather than blaming a data
cluster. Gradian cross-references the two and emits
`attribution.cluster_matches_data_defect` when the harmful cluster coincides with a mechanical
defect, that finding is Gradian working as intended.

### Experiment C: recovery

Round 1 diagnoses, round 2 retrains without the accused cluster. A pass needs
`round2_delta > round1_delta`. Also check `removal_precision` in `verdict.json`: what fraction of
the removed examples were genuinely planted. High recovery with low precision means Gradian removed
a lot of innocent data to get there.

---

## 3. Four failures from the CPU build worth knowing before you start

These are why the suite has the guards it has. All of them are easy to hit, and all of them look like
"the tool is broken".

### 3.1 The query gradient must be *contrastive* for substitution errors

This is the most important finding of the build, and it changed the code.

Plant "the capital of France is Berlin", fine-tune, and build the query gradient the textbook way,
cross-entropy on the *correct* answer over the items that regressed. The poisoned examples come out
as the **most helpful data in the corpus**:

| query gradient | planted examples | median percentile |
|---|---|---|
| CE on correct answer | **+5143** (helpful) | 93.8 |
| CE on the wrong answer | +6860 | 93.8 |
| **contrastive: correct − wrong** | **−1717** (harmful) | **5.5** |

We verified the wrong-sign result is invariant to gradient normalization (`none`/`l2`), which
modules are indexed (`down_proj`/all), the engine (DataInf/graddot), and the damping
(λ scale 0.1/0.01/0.001): 16 configurations, byte-identical conclusion. So it is not a tuning
problem.

The cause: what a fine-tune mostly teaches is the answer *format*, "when asked for a capital, emit
a bare city name". The poisoned examples teach that format perfectly, which genuinely lowers the
cross-entropy of "Paris" too. Token identity is a small part of the gradient, so alignment is
dominated by the shared task direction and the sign inverts.

Subtracting the gradient of what the model *actually said* cancels the shared direction. Gradian now
does this by default (`evaluation.query_signal: auto` → contrastive whenever generations are
available). It needs a text metric (`exact_match`/`contains`) to have a generation to contrast
against. With `loglik` there is nothing to subtract and it falls back to the classic form.

This is the reference doc's section 5 "match the loss to the scorer", one level deeper: for a
substitution error the scorer is contrastive, so the query gradient must be too.

### 3.2 The eval metric must measure correctness, not format

The same experiment with `metric="loglik"` reports the capability as **improved** (delta +3.77)
while the model confidently answers with the wrong city, because fine-tuning raised the likelihood
of the answer format, which swamps the content. Use `exact_match`/`contains` for factual
capabilities. Keep log-likelihood for capabilities with no discrete scorer, and expect a blurrier
signal. Gradian stamps `proxy_used: true` on those results.

### 3.3 A base model with no chat template silently trains on the prompt

Our first injection test used `SmolLM2-135M` (base, not `-Instruct`). It has no chat template, so
Gradian correctly falls back to supervising the whole sequence, and then the fine-tune mostly
learns formatting, no regression is measurable, and the experiment is vacuous. Gradian reports this
as `data.no_chat_template`, and the acceptance test now refuses to run without a template. Use
`-Instruct` models, or set `tokenizer.chat_template` explicitly.

### 3.4 Long examples need span-level drill-down, with a per-item query

A wrong fact buried inside a long entry is a different problem from a wrong short answer, and we
measured both halves of it.

Setup: 48 long travel-guide entries (~110 supervised tokens each) that name the **wrong** capital
once, mid-paragraph, alongside 16 short entries stating the **correct** capital, plus unrelated
filler.

**Whole-example influence gets the direction right but is heavily diluted:**

| entries | n | mean score | signed sum | rank |
|---|---|---|---|---|
| long, wrong fact buried | 48 | −0.076 (harmful) | −3.67 | most harmful cluster |
| short, correct fact | 16 | +57.96 (helpful) | +927 | most helpful |
| unrelated filler | 24 | +1.1 | +26 | n/a |

The long entries are correctly identified as the only harmful cluster, but they score ~760× weaker
*per example* than a short entry stating the same fact, because the wrong fact is ~2 of ~110
supervised tokens and the rest of the document is benign, on-topic prose. Cluster aggregation
rescued it here (48 entries summing to a clear #1). A **single** long entry with a buried error
would sit below the noise floor.

(Also worth knowing: the model never learned the wrong capital in this run. The short correct entry
won, the delta was +0.06 and the gate refused to attribute it. Buried wrong facts do not
automatically propagate.)

**Span-level drill-down fixes the resolution, if the query is per-item:**

| query used for the drill-down | culprit sentence ranked #1 |
|---|---|
| averaged over all 16 eval items | ~33% (2 of 6) |
| built from the ONE failing item | **92% (44 of 48)** |

With a per-item query the separation is not marginal: the culprit sentence scores −545 while its
neighbours sit at ±4. With an averaged query the culprit frequently scores *positive*, because the
average carries every other item's contrast direction ("Paris up, Berlin down" pollutes the scoring
of a document about Japan).

**The mirror case is much easier, and that asymmetry is the practical takeaway.** Swap the roles
(correct capital buried in the 48 long entries, wrong capital in the 16 short ones):

| entries | n | mean score | signed sum | median pct | outcome |
|---|---|---|---|---|---|
| short, wrong fact | 16 | **−55.29** | −884.6 | **8.5%** | ranked #1, **96.2%** of negative influence |
| long, correct fact buried | 48 | +0.158 (helpful) | +7.6 | 57.4% | correctly not blamed |
| unrelated filler | 24 | −3.10 | −74.5 | 23.3% | n/a |

Here the wrong info *did* propagate: delta −0.8125, significant, **14 of 16 eval items flipped**. 16
short wrong entries beat 48 long correct ones, because a short entry's gradient is concentrated on
exactly the queried token.

So the difficulty depends entirely on where the error sits, not on how much of it there is:

| error location | per-example score | detectability |
|---|---|---|
| short, direct entry | −55 | trivial, top of the ranking, ~350× stronger |
| buried in a long entry | −0.076 | needs cluster aggregation, or span drill-down for a single entry |

A corollary worth internalizing: a wrong fact stated *directly* in a handful of examples does more
damage than the same fact buried in three times as many long ones, and is far easier to find.

Use `gradian.query.spans.explain_example()`, which builds the per-item contrastive query for you.
Calling `score_example_spans()` directly means you own that rule.

---

## 4. Beyond the suite: what only a GPU can establish

The suite covers the practical claims. These are the deeper validations from doc section 6
worth running once on real hardware:

**LOO correlation (fidelity).** The direct measure of how well influence approximates
leave-one-out retraining. Feasible on a small model: retrain N times, each omitting one cluster,
and correlate the true capability change against Gradian's predicted
`sum(predicted_loss_delta_if_removed)` for that cluster. Report Spearman. With 10 clusters that is
11 LoRA runs, an hour on a 1B model, and the single most convincing number you can publish.

**EK-FAC cross-check.** `pip install -e ".[ekfac]"`, then register a kronfluence-backed engine and
compare rankings (`gradian.query.scorer.rank_correlation`). The registry and the agreement check
already exist, the engine is the remaining work. High agreement is a confidence signal, divergence
flags a shaky diagnosis (doc section 6).

**Projection sweep.** Run experiment A at `--projection-dim` ∈ {512, 2048, 8192} and against exact
gradients, and record rank correlation and store size. That gives you the accuracy/size curve the
doc asks you to pick `k` from, measured on your models rather than assumed.

**Damping sweep.** λ is load-bearing (doc section 7): too small and the inverse curvature
explodes on near-zero eigenvalues, too large and it collapses toward raw gradient dot products.
Tune
`engine.lambda_scale` against the LOO correlation above, and log it. A λ change is a result change,
which is why it is stamped into every index manifest.

---

## 5. If something fails

- **`no LoRA blocks found`**: the adapter is not attached, or `--module-filter` matched nothing.
  Check `gradian show-index <dir>` for the block list.
- **OOM during indexing**: lower `--grad-batch-size` first, then `--max-seq-length`. Those two are
  the only levers that touch the allocation that actually fails. The peak is the fp32 upcast of the
  logits in `grads/loss.py:per_example_ce`, sized `grad_batch_size × supervised_tokens × vocab × 4`
  bytes (only the positions the loss is computed on reach that upcast), and with a 128k-vocab
  Llama-3 tokenizer at batch 8 and 512 supervised tokens that single tensor is about 2.1 GB
  on top of the fp16 logits it came from. It exists because Gradian computes its own loss from
  logits instead of passing `labels=`, which is what keeps fused cross-entropy kernels from
  swallowing the gradients we capture, so it is a deliberate cost rather than a leak.

  `--fp16-store`, `--module-filter` and `--projection-dim` shrink the *stored index*, not this
  tensor, so they will not rescue an OOM at this line. Reach for them when the index does not fit
  on disk. Measured: a 16GB V100 OOMs at the default `--grad-batch-size 8` on a 1B model and runs
  clean at 2.
- **`unsloth` unavailable despite being installed**: `gradian doctor` will say why, usually no
  visible CUDA device, or a version outside the pin window.
- **`'LlamaAttention' object has no attribute 'apply_qkv'`**: `FastLanguageModel.from_pretrained`
  rebinds `forward` on the transformers attention, decoder-layer and model classes process-wide,
  and those fast paths read per-instance attributes that unsloth installs only on models it builds
  itself. Anything built afterwards by plain transformers in the same process inherits the fast
  forward without them, which is what `analyze()` does when it reloads a checkpoint to index
  gradients. `UnslothBackend.train()` now snapshots those `forward` attributes and restores them
  when training finishes, so train-then-analyze works in one process. If you meet this error
  outside `gradian train`, you imported unsloth and built a model with it yourself: either do the
  analysis in a separate process, or call
  `gradian.model.backends.unsloth._restore_modeling_forwards()` with your own snapshot.
- **Gradients differ between the unsloth and HF backends**: run
  `pytest tests/integration/test_unsloth_contract.py -q` first. The compat shim
  (`gradian.model.backends.unsloth.prepare_for_gradients`) handles fused cross-entropy, gradient
  checkpointing, `use_cache`, and inference mode. If parity fails on real hardware, that function
  is where to look, and `unsloth_compat_report(model)` prints the state a reviewer needs.
- **Everything passes but the report looks wrong**: read the Provenance section of the markdown
  report. Every number is reproducible from the manifest it prints, and a changed manifest field is
  the explanation.