Source: https://gradian.dev/docs/configuration

# Configuration

Anything that changes a number lives in the config and gets hashed into the index id, so "why did my
scores change?" always has an answer: something in this object changed.

Write a starter file, then edit it:

```bash
gradian init-config --base-model meta-llama/Llama-3.2-1B-Instruct
```

Unknown keys are rejected rather than ignored, so a typo fails loudly instead of silently doing
nothing.

## A realistic config

```yaml
seed: 1234
base_model: meta-llama/Llama-3.2-1B-Instruct
adapter_path: runs/my-finetune/adapter
dataset: train.jsonl
eval_dataset: capability_eval.jsonl
run_dir: runs/my-finetune

grads:
  batch_size: 2            # lower this first if you run out of GPU memory
  max_seq_length: 1024
  module_filter: down_proj # down_proj is the largest single site; measured ~4.3x smaller
  storage_dtype: float16   # halves the index on disk

engine:
  name: datainf
  lambda_scale: 0.1

evaluation:
  metric: exact_match      # not loglik, for factual capabilities
  n_bootstrap: 2000

clustering:
  method: auto
  n_clusters: 8
```

## Top level

| Field | Type | Default | What it does |
|---|---|---|---|
| `seed` | int | `1234` | Seeds everything. Part of the index id, so changing it invalidates the cache |
| `device` | str | `auto` | `auto` picks CUDA when visible, else CPU. Set `mps` explicitly on Apple Silicon, which is never auto-selected |
| `dtype` | str | `auto` | Model dtype |
| `base_model` | str | empty | HuggingFace id or local path you fine-tuned from |
| `tokenizer` | str or null | `null` | Override the tokenizer. Defaults to the base model's |
| `adapter_path` | str or null | `null` | PEFT LoRA adapter directory. Required for attribution |
| `dataset` | str or null | `null` | Training data, `.jsonl`, `.json` or a `save_to_disk` directory |
| `eval_dataset` | str or null | `null` | Items describing the capability, with the answers you wanted |
| `run_dir` | str or null | `null` | Run directory with trainer config and logs, which the audit reads |
| `cache_dir` | str | `.gradian` | Where the content-addressed index cache lives |
| `trust_remote_code` | bool | `false` | Passed through to transformers. Leave off unless you need it |
| `attn_implementation` | str | `eager` | Forced to `eager` by default, because the attention implementation has to be hookable and deterministic |

## grads

Per-example gradient extraction. These are the settings that decide how long indexing takes and how
much disk and GPU memory it needs.

| Field | Type | Default | What it does |
|---|---|---|---|
| `strategy` | `hooks`, `loop`, `vmap` | `hooks` | How per-example gradients are captured. `hooks` is exact and the default. `loop` is the batch-size-1 reference. `vmap` is brittle over transformers forwards and opt-in only |
| `loss_reduction` | `mean`, `sum` | `mean` | Per-example loss reduction over supervised tokens |
| `normalization` | `none`, `l2`, `token_count` | `none` | Optional gradient normalization before scoring |
| `batch_size` | int | `4` | **The main memory lever.** The logit upcast costs `batch_size x supervised_tokens x vocab x 4` bytes -- the largest allocation you control, though not the whole peak, since weights and activations also count (measured on a 1B model at batch 2 / 512 tokens, 80% prompt: 3.25 GB peak against 0.10 GB of upcast). Lower this first on an out-of-memory error |
| `max_seq_length` | int | `1024` | Truncation length during indexing. The second memory lever |
| `module_filter` | str or null | `null` | Regex on the short module name. `down_proj` indexes one module type instead of seven. Since it is the largest single site rather than an equal 1/7th share, measured savings are about 4.3x, not 7x |
| `block_granularity` | `module`, `matrix` | `module` | Whether a block is a whole module or each of A and B separately |
| `projection_dim` | int or null | `null` | Johnson-Lindenstrauss projection width. Makes storage independent of model size, at a measured cost in ranking fidelity (Spearman 0.6 to 0.9) |
| `projection_kind` | `countsketch`, `rademacher` | `countsketch` | Projection family. Countsketch is unbiased in expectation only |
| `projection_seed` | int | `0` | Seeds the projection, and is hashed into the index id |
| `storage_dtype` | `float32`, `float16` | `float32` | `float16` halves the index on disk, nearly free in accuracy |
| `chunk_size` | int | `256` | Examples per write chunk in the memmapped store |

`module_filter`, `projection_dim` and `storage_dtype` shrink the **stored index**. `batch_size` and
`max_seq_length` shrink the **peak GPU allocation**. They are different problems, so reaching for the
wrong group will not rescue an out-of-memory error. See
[Troubleshooting](/docs/troubleshooting#out-of-memory-during-indexing).

Only the positions the loss is computed on reach the fp32 logit upcast, so the peak tracks
supervised tokens rather than sequence length. Prompt tokens and padding cost nothing there, which
is why a run with long answers can need more memory than a run with the same `max_seq_length` and
short ones.

`module_filter` matches the full short module name, which looks like
`model.layers.12.mlp.down_proj`, so it selects layers as readily as module types. Pinning the layer
number in the pattern indexes only the layers you name, and at depth that cuts more than choosing a
module type does, because a model has many more layers than module types.

## engine

| Field | Type | Default | What it does |
|---|---|---|---|
| `name` | str | `datainf` | `datainf`, `graddot` or `tracin` |
| `lambda_scale` | float | `0.1` | Damping, relative to the mean squared gradient norm per block. Load-bearing: too small and the inverse curvature explodes on near-zero eigenvalues, too large and it collapses toward raw gradient dot products |
| `lambda_override` | float or null | `null` | Set damping absolutely instead of by scale |
| `normalize_query` | bool | `false` | Applies to the curvature-free engines only |

A change to `lambda_scale` is a change to your results. It is hashed into the index id and printed in
every report manifest for exactly that reason.

## evaluation

| Field | Type | Default | What it does |
|---|---|---|---|
| `max_new_tokens` | int | `32` | Generation cap when measuring the capability. Raise it if your expected answers are longer |
| `batch_size` | int | `8` | Generation batch size |
| `n_bootstrap` | int | `2000` | Bootstrap resamples for the confidence interval |
| `ci_level` | float | `0.95` | Confidence level for the significance gate |
| `metric` | `exact_match`, `contains`, `not_contains`, `loglik` | `exact_match` | How capability is measured. Use `exact_match` or `contains` for factual capabilities |
| `greedy` | bool | `true` | Greedy decoding, for reproducibility |
| `require_significant` | bool | `true` | Refuse to attribute when the delta's confidence interval includes zero. Turning this off removes the main guard against reading noise |
| `query_signal` | `auto`, `contrastive`, `target` | `auto` | How the query gradient is built. See below |

`query_signal` is worth understanding rather than guessing:

- `contrastive` is the gradient of the expected answer minus the gradient of what the model actually
  said. Required for substitution errors, where the model now says something else, because the
  gradient of the expected answer alone is dominated by answer format and **inverts the sign of the
  diagnosis**.
- `target` is the gradient of the expected answer only. The classic formulation, and correct when the
  failure is "the right answer got less likely" rather than "it now says something else".
- `auto` uses contrastive whenever generations are available, meaning a text metric, and falls back to
  target otherwise.

Because `loglik` produces no generated text, pairing it with `auto` silently gives you `target`. That
combination is the single easiest way to get a confidently backwards answer.

## clustering

| Field | Type | Default | What it does |
|---|---|---|---|
| `method` | `auto`, `hdbscan`, `kmeans`, `declared`, `none` | `auto` | `declared` uses the `cluster` or `cluster_id` field on your rows instead of clustering |
| `n_clusters` | int | `8` | Cluster count for KMeans |
| `min_cluster_size` | int | `5` | Minimum cluster size for HDBSCAN |
| `embed_model` | str | `sentence-transformers/all-MiniLM-L6-v2` | Embedding model for semantic clustering |
| `embed_backend` | `auto`, `sentence-transformers`, `tfidf` | `auto` | Falls back to TF-IDF when the `cluster` extra is not installed |
| `max_features` | int | `4096` | TF-IDF vocabulary cap |
| `svd_dim` | int | `64` | SVD width in the TF-IDF fallback path |
| `top_examples_per_cluster` | int | `5` | How many example excerpts each cluster shows in the report |
| `label_top_terms` | int | `4` | How many terms make up a cluster's label |

If your data already has a taxonomy, set `method: declared` and put `cluster_id` on each row. The
report then speaks in your categories rather than discovered ones, which usually makes the verdict
directly actionable.

## train

Settings for the optional `gradian train` path. These do not affect attribution.

| Field | Type | Default |
|---|---|---|
| `backend` | `hf`, `unsloth` | `hf` |
| `base_model` | str | empty |
| `output_dir` | str | `runs/run` |
| `r` | int | `8` |
| `lora_alpha` | int | `16` |
| `lora_dropout` | float | `0.0` |
| `target_modules` | list of str or null | `null` |
| `learning_rate` | float | `0.0002` |
| `num_train_epochs` | float | `1.0` |
| `per_device_train_batch_size` | int | `2` |
| `gradient_accumulation_steps` | int | `1` |
| `warmup_ratio` | float | `0.03` |
| `lr_scheduler_type` | str | `linear` |
| `max_seq_length` | int | `512` |
| `logging_steps` | int | `1` |
| `save_steps` | int | `0` |
| `save_checkpoints` | bool | `false` |
| `max_grad_norm` | float | `1.0` |
| `weight_decay` | float | `0.0` |
| `load_in_4bit` | bool | `false` |
| `extra` | dict | `{}` |

Two notes. `backend: trl` is not a valid value here even though `--backend trl` works on the command
line. And `load_in_4bit` affects training only, since the attribution loader has no quantization path,
so a 4-bit fine-tune is analyzed at full precision.

`lora_dropout` above zero is fine for training, and dropout is always disabled during attribution,
because otherwise per-example gradients would not be reproducible.