Source: https://gradian.dev/docs/limitations # Limitations Some of these decide whether Gradian fits your setup at all, so it is worth checking against your setup before you invest time in a fine-tune, whichever point in these docs you found this page from. ## What Gradian will not analyze **LoRA adapters only.** Attribution differentiates with respect to adapter parameters, so a full fine-tune has nothing to attribute over and `gradian attribute` exits rather than guess. The hook path covers LoRA on the `Linear` layers of a HuggingFace causal LM. LoRA on embeddings or convolutions is skipped, so an adapter that lives only there produces no blocks to index. **Causal language models only.** Models are loaded as `AutoModelForCausalLM` with a `CAUSAL_LM` PEFT task type. Sequence classification and encoder-decoder adapters are not supported. **A 4-bit adapter is analyzed against an unquantized base.** The attribution loader has no quantization path, so a QLoRA fine-tune indexes at full precision. That costs memory, and it moves the loss slightly relative to how the adapter was trained. **A base model without a chat template degrades quietly.** Supervision falls back to the whole sequence rather than the assistant turn. The `data.no_chat_template` check catches this, which is one reason to run `gradian diagnose` first. ## Scale and hardware **One device, one process.** No `device_map`, no DDP or FSDP, no sharding. The device is CUDA or CPU, chosen automatically. MPS works if you set it explicitly and is never auto-selected. Windows needs WSL2, and Intel Macs are not supported. **The gradient index is big.** It holds one vector per training example. | Model | LoRA r=16, all 7 sites | Per example | 2000 examples | |---|---|---|---| | 1B | about 11M params | about 45 MB | about 90 GB | | 8B | about 42M params | about 170 MB | about 340 GB | Three settings shrink the stored index: `grads.module_filter` cuts it about 4.3x, measured, by indexing one module type (`down_proj`) instead of seven -- it is the largest single site, not an equal 1/7th share, so the saving is smaller than a naive 7x -- `grads.storage_dtype: float16` halves it, and `grads.projection_dim` makes it independent of model size at a measured cost in ranking fidelity, with Spearman against exact scores falling to roughly 0.6 to 0.9 depending on the width you pick. **Peak GPU memory is the logit upcast, not the index.** The allocation that actually fails is `grads.batch_size x supervised_tokens x vocab x 4` bytes, from the fp32 upcast of the logits. Only the positions the loss is computed on reach that upcast, so prompt tokens and padding are free, but a batch with long answers costs more than a batch of the same length with short ones. With a 128k-vocab Llama-3 tokenizer at batch 8 and 512 supervised tokens that single tensor is about 2.1 GB on top of the fp16 logits it came from. It exists because Gradian computes its own loss from logits instead of passing `labels=`, which is what keeps fused cross-entropy kernels from swallowing the gradients it captures, so it is a deliberate cost rather than a leak. Only `grads.batch_size` and `grads.max_seq_length` reduce it. The three index-shrinking settings above will not rescue an out-of-memory error at this line. Measured: a 16GB V100 runs out of memory on a 1B model at a gradient batch of 8, and runs clean at 2. ## Inputs **Datasets are local.** A `.jsonl` file, a `.json` file, or a directory written by HuggingFace `save_to_disk`. Hub dataset ids are not resolved for you. **Dataset order matters.** Example ids embed position, and the index cache key derives from them, so a reordered file is a different dataset as far as the cache is concerned. **Without an eval set you get a smoke test.** Attribution falls back to scoring against the first few training examples and flags `attribution.no_capability_target`. It runs, but it is not a diagnosis. ## Method **Influence is a first-order approximation, not a proof.** A score predicts what removing an example would do to the loss. The only thing that demonstrates it is dropping the accused examples and retraining. **Plenty of honest runs end without a verdict.** If the capability delta's bootstrap confidence interval includes zero, Gradian reports the delta and explicitly declines to attribute. Small eval sets land there often. This is deliberate, and turning off `evaluation.require_significant` removes the main guard against reading noise as a result. **The capability metric is load-bearing.** A log-likelihood metric can go up even under a successful poisoning, because a fine-tune mostly teaches answer format. It also produces no generated text for the contrastive query to subtract, so it silently falls back to the classic formulation, which inverts the sign on substitution failures. Use `exact_match` or `contains` for factual capabilities. **A single error buried in a long document can sit below the noise floor.** Dilution is roughly 760x against a short entry stating the same thing. Cluster aggregation recovers it when several examples share the problem, and span-level drill-down localizes it 92% of the time, but only when the query is built from the single failing item. **Assumes a converged optimum.** Influence functions are defined at a minimum. Mid-schedule checkpoints give noisier rankings. **The base model stays opaque.** Gradian runs the base model and bounds blame with base scores and preference margins, but it never sees the base model's training data and makes no claim about it. **Determinism is enforced during extraction, and checkable afterwards.** This paragraph used to say reruns were "in practice stable on GPU". They were not. Deterministic algorithms were requested in `warn_only` mode, and the cuBLAS workspace variable was set after the process had already created cuBLAS handles, so nondeterministic reduction order got through as a warning. Two builds of one configuration on one machine differed in 99.99978% of stored entries, and a score vector reproduced at only Spearman 0.781 on byte-identical inputs. Extraction now runs under strict deterministic kernels: an op with no deterministic implementation raises and names itself rather than drifting quietly. `grads.deterministic: false` restores the old behavior for stacks where some op cannot comply, and the choice is hashed into the index id either way. The residual limitation is that an index built elsewhere, or before this changed, cannot be trusted on its id alone. `index_id` hashes the computation's *inputs*, so two builds that produced different numbers collide on one id. Every manifest therefore also records a blake2b digest per stored block and one combined `content_digest`, fingerprinting the *outputs*. Run `gradian index-verify ` to check the bytes on disk against them: exit `0` verified, `1` contents differ, `2` the index predates digests and cannot be verified at all. ## Project status **Alpha.** Version 0.1.0, classified as `Development Status :: 3 - Alpha`, installed from the script or a checkout. There is no package-index release, so installing builds from a tag or sha in the public repository. That is reproducible, but slower than a wheel. **Gradian does not launch unsloth training.** It analyses the adapters unsloth produces, which is the part that matters and which needs nothing from unsloth: fine-tune however you like, then point `gradian attribute` at the adapter directory. There is no `--backend unsloth`, and asking for one says this rather than failing obscurely. The reason is that `import unsloth` patches transformers, peft and torch process-wide, mostly at import time, and attribution is what pays for it. Snapshotting those bindings and putting them back does not work. Measured on a V100 after a full restore, 332 unsloth-owned functions were still bound inside transformers and at least one attention registry was mutated in place rather than rebound, which no snapshot of the original objects can detect, and the attribution forward pass still died inside eager attention on a mask sized for the prompt against scores sized for the whole sequence. Unsloth's own maintainers give the same answer, which is a separate process. So `load_for_attribution` refuses up front in any process that has imported unsloth, naming both ways out. What that means in practice: - **Fine-tuned with unsloth already?** Nothing to do. Your adapter is a standard PEFT directory and Gradian reads it with plain transformers and peft. - **Want Gradian to run the training** (counterfactual retraining, the injection test)? Use `--backend hf`, the default. - **Holding an unsloth model object in memory** and set on attributing in that same process? Pass it through `gradian.model.unsloth_compat.prepare_for_gradients`, wrap it in a `LoadedModel`, and hand that to `build_index`, so no plain model is ever built. This path is verified at contract level on CPU and has not been run end to end on a GPU. One thing this costs: whether an unsloth fine-tune and an HF one of the same data attribute identically is unproven here, and cannot be proven in one process. **Two-stage retrieval can drop a true positive before anything is scored.** With `grads.shortlist_fraction` set, a representation filter chooses which examples get a gradient at all, and whatever it discards is never ranked. It is off by default. See [Configuration](/docs/configuration) for the measured recall. The repository README has a "What is next" section covering the work that addresses several of these.