Source: https://gradian.dev/docs/limitations

# Limitations

Some of these decide whether Gradian fits your setup at all, so it is worth checking against your
setup before you invest time in a fine-tune, whichever point in these docs you found this page from.

## What Gradian will not analyze

**LoRA adapters only.** Attribution differentiates with respect to adapter parameters, so a full
fine-tune has nothing to attribute over and `gradian attribute` exits rather than guess. The hook path
covers LoRA on the `Linear` layers of a HuggingFace causal LM. LoRA on embeddings or convolutions is
skipped, so an adapter that lives only there produces no blocks to index.

**Causal language models only.** Models are loaded as `AutoModelForCausalLM` with a `CAUSAL_LM` PEFT
task type. Sequence classification and encoder-decoder adapters are not supported.

**A 4-bit adapter is analyzed against an unquantized base.** The attribution loader has no
quantization path, so a QLoRA fine-tune indexes at full precision. That costs memory, and it moves the
loss slightly relative to how the adapter was trained.

**A base model without a chat template degrades quietly.** Supervision falls back to the whole
sequence rather than the assistant turn. The `data.no_chat_template` check catches this, which is one
reason to run `gradian diagnose` first.

## Scale and hardware

**One device, one process.** No `device_map`, no DDP or FSDP, no sharding. The device is CUDA or CPU,
chosen automatically. MPS works if you set it explicitly and is never auto-selected. Windows needs
WSL2, and Intel Macs are not supported.

**The gradient index is big.** It holds one vector per training example.

| Model | LoRA r=16, all 7 sites | Per example | 2000 examples |
|---|---|---|---|
| 1B | about 11M params | about 45 MB | about 90 GB |
| 8B | about 42M params | about 170 MB | about 340 GB |

Three settings shrink the stored index: `grads.module_filter` cuts it about 4.3x, measured, by
indexing one module type (`down_proj`) instead of seven -- it is the largest single site, not an
equal 1/7th share, so the saving is smaller than a naive 7x -- `grads.storage_dtype: float16`
halves it, and `grads.projection_dim` makes it
independent of model size at a measured cost in ranking fidelity, with Spearman against exact scores
falling to roughly 0.6 to 0.9 depending on the width you pick.

**Peak GPU memory is the logit upcast, not the index.** The allocation that actually fails is
`grads.batch_size x supervised_tokens x vocab x 4` bytes, from the fp32 upcast of the logits. Only
the positions the loss is computed on reach that upcast, so prompt tokens and padding are free, but
a batch with long answers costs more than a batch of the same length with short ones. With a
128k-vocab Llama-3 tokenizer at batch 8 and 512 supervised tokens that single tensor is about 2.1 GB
on top of the fp16 logits it came from. It exists because Gradian computes its own loss from logits
instead of passing `labels=`, which is what keeps fused cross-entropy kernels from swallowing the
gradients it captures, so it is a deliberate cost rather than a leak.

Only `grads.batch_size` and `grads.max_seq_length` reduce it. The three index-shrinking settings above
will not rescue an out-of-memory error at this line. Measured: a 16GB V100 runs out of memory on a 1B
model at a gradient batch of 8, and runs clean at 2.

## Inputs

**Datasets are local.** A `.jsonl` file, a `.json` file, or a directory written by HuggingFace
`save_to_disk`. Hub dataset ids are not resolved for you.

**Dataset order matters.** Example ids embed position, and the index cache key derives from them, so a
reordered file is a different dataset as far as the cache is concerned.

**Without an eval set you get a smoke test.** Attribution falls back to scoring against the first few
training examples and flags `attribution.no_capability_target`. It runs, but it is not a diagnosis.

## Method

**Influence is a first-order approximation, not a proof.** A score predicts what removing an example
would do to the loss. The only thing that demonstrates it is dropping the accused examples and
retraining.

**Plenty of honest runs end without a verdict.** If the capability delta's bootstrap confidence
interval includes zero, Gradian reports the delta and explicitly declines to attribute. Small eval sets
land there often. This is deliberate, and turning off `evaluation.require_significant` removes the main
guard against reading noise as a result.

**The capability metric is load-bearing.** A log-likelihood metric can go up even under a successful
poisoning, because a fine-tune mostly teaches answer format. It also produces no generated text for the
contrastive query to subtract, so it silently falls back to the classic formulation, which inverts the
sign on substitution failures. Use `exact_match` or `contains` for factual capabilities.

**A single error buried in a long document can sit below the noise floor.** Dilution is roughly 760x
against a short entry stating the same thing. Cluster aggregation recovers it when several examples
share the problem, and span-level drill-down localizes it 92% of the time, but only when the query is
built from the single failing item.

**Assumes a converged optimum.** Influence functions are defined at a minimum. Mid-schedule
checkpoints give noisier rankings.

**The base model stays opaque.** Gradian runs the base model and bounds blame with base scores and
preference margins, but it never sees the base model's training data and makes no claim about it.

**Determinism is best-effort on GPU.** Deterministic algorithms are requested with `warn_only`, so a
non-deterministic kernel warns rather than fails. Reruns are bit-identical on CPU, and in practice
stable on GPU, but the guarantee is not absolute.

## Project status

**Alpha.** Version 0.1.0, classified as `Development Status :: 3 - Alpha`, installed from the script
or a checkout. There is no package-index release, so installing needs access to the repository.

**The unsloth training backend is verified at contract level on CPU only.** End-to-end parity between
the HF and unsloth backends is GPU-gated and has no recorded pass yet. This affects `gradian train
--backend unsloth`, not analysis of a model you trained with unsloth yourself.

The repository README has a "What is next" section covering the work that addresses several of
these, including the EK-FAC cross-check engine and leave-one-out fidelity correlation.