Source: https://gradian.dev/docs/troubleshooting # Troubleshooting Start with `gradian doctor`. It reports installed versions against the supported window, visible devices, determinism settings, and which backends are available along with the reason any are not. Its output is also the right thing to paste into an issue. ## `no LoRA blocks found` The adapter is not attached, or `grads.module_filter` matched nothing. Gradian attributes over adapter parameters, so with no LoRA blocks there is nothing to differentiate. Check that `--adapter` points at a PEFT adapter directory, and if you set a module filter, that its regex actually matches your module names. `gradian show-index ` prints the block list from a built index. Also worth knowing: LoRA on embeddings or convolutions is skipped by the hook path, so an adapter that targets only those produces no blocks. ## Out of memory during indexing Lower `grads.batch_size` first, then `grads.max_seq_length`. Those two are the only settings that touch the allocation that actually fails. The peak is the fp32 upcast of the logits, sized `batch_size x supervised_tokens x vocab x 4` bytes. Gradian gathers the supervised positions before that upcast, so prompt tokens and padding cost nothing here. On instruction data where the prompt is most of the sequence that is a large reduction, and it also means two runs at the same `max_seq_length` can differ in peak memory when one has longer answers than the other. With a 128k-vocab Llama-3 tokenizer at batch 8 and 512 supervised tokens that single tensor is about 2.1 GB on top of the fp16 logits it came from. `grads.storage_dtype`, `grads.module_filter` and `grads.projection_dim` shrink the **stored index**, not this tensor, so they will not rescue an out-of-memory error at this line. Reach for them when the index does not fit on disk. Measured: a 16GB V100 runs out of memory at a gradient batch of 8 on a 1B model, and runs clean at 2. ## The index does not fit on disk That is the other problem, and it has different levers. ```yaml grads: module_filter: down_proj # about 4.3x smaller storage_dtype: float16 # halves it, nearly free in accuracy projection_dim: 2048 # independent of model size, costs ranking fidelity ``` Attribution is over the parameters you index, so a subset is a legitimate and documented choice rather than a workaround. The projection is the one with a real accuracy cost, so tune it rather than assuming a width. `module_filter` can also cut by depth. It is a regex over the full short module name, which looks like `model.layers.12.mlp.down_proj`, so a pattern that pins the layer number indexes only the layers you name. At depth that is a sharper lever than module type, because the count of layers is larger than the count of module types. ## `attribution.no_capability_target` in the report You ran without `--eval-dataset`, so Gradian fell back to attributing against the first few training examples. It runs, but it is not a diagnosis. Give it an eval set describing the capability, with the answers you wanted. ## The report says the delta is not significant, and refuses to attribute Working as intended. The capability delta's bootstrap confidence interval includes zero, so there is no established regression to explain. Usually the eval set is too small. Widen it. If you are certain the regression is real, check that `evaluation.metric` suits the capability, since an `exact_match` metric on free-form answers will measure formatting rather than correctness. Do not reach for `evaluation.require_significant: false`. That removes the main guard against reading noise as a result. ## The results look backwards, and the poisoned data ranks as helpful Almost always the query gradient. For a substitution failure, where the model now says something else, the gradient of the expected answer alone is dominated by answer format, and it inverts the sign of the diagnosis. Check two settings. `evaluation.query_signal` should be `contrastive` or `auto`, and `evaluation.metric` should be a text metric. Because `loglik` produces no generated text to subtract, pairing it with `auto` silently gives you the classic formulation and the backwards answer along with it. ## The culprit is a single sentence and nothing ranks it Whole-example influence dilutes a wrong sentence inside a long entry by roughly 760x. Cluster aggregation recovers it when several examples share the problem. For a single entry, use span-level drill-down through `gradian.query.spans`, and build the query from the one failing item rather than an average, which is the difference between 92% and 33% localization. ## `unsloth` unavailable despite being installed `gradian doctor` will say why. Usually there is no visible CUDA device, or the installed version is outside the pinned window. unsloth cannot be installed on a CPU-only machine at all. ## `'LlamaAttention' object has no attribute 'apply_qkv'` `FastLanguageModel.from_pretrained` rebinds `forward` on the transformers attention, decoder-layer and model classes process-wide, and those fast paths read per-instance attributes that unsloth installs only on models it builds itself. Anything built afterwards by plain transformers in the same process inherits the fast forward without them. `UnslothBackend.train()` snapshots those `forward` attributes and restores them when training finishes, so train-then-analyze works in one process. If you hit this outside `gradian train`, you imported unsloth and built a model with it yourself. Either do the analysis in a separate process, or call `gradian.model.backends.unsloth._restore_modeling_forwards()` with your own snapshot. ## Gradients differ between the unsloth and HF backends Run `pytest tests/integration/test_unsloth_contract.py -q` first. The compat shim `gradian.model.backends.unsloth.prepare_for_gradients` handles fused cross-entropy, gradient checkpointing, `use_cache` and inference mode. If parity fails on real hardware, that function is where to look, and `unsloth_compat_report(model)` prints the state a reviewer needs. ## Versions drift out of the supported window `gradian doctor` warns rather than fails. The supported window is torch 2.4 to 2.11, transformers 4.51 to 5.5, peft 0.18 and above, and trl 0.24 and below. The exact pins live in `constraints/`. Installing with the script keeps you inside the window automatically. ## Everything passes but the report looks wrong Read the Provenance section of the markdown report. Every number is reproducible from the manifest it prints, and a changed manifest field is the explanation. `gradian show-index ` prints the same manifest for an index directory. If two runs disagree, diff their manifests before anything else. The index id hashes the checkpoint, dataset, gradient spec, seed, damping and library versions, so a differing id means one of those changed.