Configuration

Every field in gradian.yaml, with its type, default and effect. Anything that changes a number lives here and is hashed into the index id.

View .md

Anything that changes a number lives in the config and gets hashed into the index id, so "why did my scores change?" always has an answer: something in this object changed.

Write a starter file, then edit it:

gradian init-config --base-model meta-llama/Llama-3.2-1B-Instruct

Unknown keys are rejected rather than ignored, so a typo fails loudly instead of silently doing nothing.

A realistic config#

seed: 1234
base_model: meta-llama/Llama-3.2-1B-Instruct
adapter_path: runs/my-finetune/adapter
dataset: train.jsonl
eval_dataset: capability_eval.jsonl
run_dir: runs/my-finetune
 
grads:
  batch_size: 2            # lower this first if you run out of GPU memory
  max_seq_length: 1024
  module_filter: down_proj # down_proj is the largest single site; measured ~4.3x smaller
  storage_dtype: float16   # halves the index on disk
 
engine:
  name: datainf
  lambda_scale: 0.1
 
evaluation:
  metric: exact_match      # not loglik, for factual capabilities
  n_bootstrap: 2000
 
clustering:
  method: auto
  n_clusters: 8

Top level#

Field	Type	Default	What it does
`seed`	int	`1234`	Seeds everything. Part of the index id, so changing it invalidates the cache
`device`	str	`auto`	`auto` picks CUDA when visible, else CPU. Set `mps` explicitly on Apple Silicon, which is never auto-selected
`dtype`	str	`auto`	Model dtype
`base_model`	str	empty	HuggingFace id or local path you fine-tuned from
`tokenizer`	str or null	`null`	Override the tokenizer. Defaults to the base model's
`adapter_path`	str or null	`null`	PEFT LoRA adapter directory. Required for attribution
`dataset`	str or null	`null`	Training data, `.jsonl`, `.json` or a `save_to_disk` directory
`eval_dataset`	str or null	`null`	Items describing the capability, with the answers you wanted
`run_dir`	str or null	`null`	Run directory with trainer config and logs, which the audit reads
`cache_dir`	str	`.gradian`	Where the content-addressed index cache lives
`trust_remote_code`	bool	`false`	Passed through to transformers. Leave off unless you need it
`attn_implementation`	str	`eager`	Forced to `eager` by default, because the attention implementation has to be hookable and deterministic

grads#

Per-example gradient extraction. These are the settings that decide how long indexing takes and how much disk and GPU memory it needs.

Field	Type	Default	What it does
`strategy`	`hooks`, `loop`, `vmap`	`hooks`	How per-example gradients are captured. `hooks` is exact and the default. `loop` is the batch-size-1 reference. `vmap` is brittle over transformers forwards and opt-in only
`loss_reduction`	`mean`, `sum`	`mean`	Per-example loss reduction over supervised tokens
`normalization`	`none`, `l2`, `token_count`	`none`	Optional gradient normalization before scoring
`batch_size`	int	`4`	The main memory lever. The logit upcast costs `batch_size x supervised_tokens x vocab x 4` bytes -- the largest allocation you control, though not the whole peak, since weights and activations also count (measured on a 1B model at batch 2 / 512 tokens, 80% prompt: 3.25 GB peak against 0.10 GB of upcast). Lower this first on an out-of-memory error
`max_seq_length`	int	`1024`	Truncation length during indexing. The second memory lever
`module_filter`	str or null	`null`	Regex on the short module name. `down_proj` indexes one module type instead of seven. Since it is the largest single site rather than an equal 1/7th share, measured savings are about 4.3x, not 7x
`block_granularity`	`module`, `matrix`	`module`	Whether a block is a whole module or each of A and B separately
`projection_dim`	int or null	`null`	Johnson-Lindenstrauss projection width. Makes storage independent of model size, at a measured cost in ranking fidelity (Spearman 0.6 to 0.9)
`projection_kind`	`countsketch`, `rademacher`	`countsketch`	Projection family. Countsketch is unbiased in expectation only
`projection_seed`	int	`0`	Seeds the projection, and is hashed into the index id
`storage_dtype`	`float32`, `float16`	`float32`	`float16` halves the index on disk, nearly free in accuracy
`chunk_size`	int	`256`	Examples per write chunk in the memmapped store

module_filter, projection_dim and storage_dtype shrink the stored index. batch_size and max_seq_length shrink the peak GPU allocation. They are different problems, so reaching for the wrong group will not rescue an out-of-memory error. See Troubleshooting.

Only the positions the loss is computed on reach the fp32 logit upcast, so the peak tracks supervised tokens rather than sequence length. Prompt tokens and padding cost nothing there, which is why a run with long answers can need more memory than a run with the same max_seq_length and short ones.

module_filter matches the full short module name, which looks like model.layers.12.mlp.down_proj, so it selects layers as readily as module types. Pinning the layer number in the pattern indexes only the layers you name, and at depth that cuts more than choosing a module type does, because a model has many more layers than module types.

engine#

Field	Type	Default	What it does
`name`	str	`datainf`	`datainf`, `graddot` or `tracin`
`lambda_scale`	float	`0.1`	Damping, relative to the mean squared gradient norm per block. Load-bearing: too small and the inverse curvature explodes on near-zero eigenvalues, too large and it collapses toward raw gradient dot products
`lambda_override`	float or null	`null`	Set damping absolutely instead of by scale
`normalize_query`	bool	`false`	Applies to the curvature-free engines only

A change to lambda_scale is a change to your results. It is hashed into the index id and printed in every report manifest for exactly that reason.

evaluation#

Field	Type	Default	What it does
`max_new_tokens`	int	`32`	Generation cap when measuring the capability. Raise it if your expected answers are longer
`batch_size`	int	`8`	Generation batch size
`n_bootstrap`	int	`2000`	Bootstrap resamples for the confidence interval
`ci_level`	float	`0.95`	Confidence level for the significance gate
`metric`	`exact_match`, `contains`, `not_contains`, `loglik`	`exact_match`	How capability is measured. Use `exact_match` or `contains` for factual capabilities
`greedy`	bool	`true`	Greedy decoding, for reproducibility
`require_significant`	bool	`true`	Refuse to attribute when the delta's confidence interval includes zero. Turning this off removes the main guard against reading noise
`query_signal`	`auto`, `contrastive`, `target`	`auto`	How the query gradient is built. See below

query_signal is worth understanding rather than guessing:

contrastive is the gradient of the expected answer minus the gradient of what the model actually said. Required for substitution errors, where the model now says something else, because the gradient of the expected answer alone is dominated by answer format and inverts the sign of the diagnosis.
target is the gradient of the expected answer only. The classic formulation, and correct when the failure is "the right answer got less likely" rather than "it now says something else".
auto uses contrastive whenever generations are available, meaning a text metric, and falls back to target otherwise.

Because loglik produces no generated text, pairing it with auto silently gives you target. That combination is the single easiest way to get a confidently backwards answer.

clustering#

Field	Type	Default	What it does
`method`	`auto`, `hdbscan`, `kmeans`, `declared`, `none`	`auto`	`declared` uses the `cluster` or `cluster_id` field on your rows instead of clustering
`n_clusters`	int	`8`	Cluster count for KMeans
`min_cluster_size`	int	`5`	Minimum cluster size for HDBSCAN
`embed_model`	str	`sentence-transformers/all-MiniLM-L6-v2`	Embedding model for semantic clustering
`embed_backend`	`auto`, `sentence-transformers`, `tfidf`	`auto`	Falls back to TF-IDF when the `cluster` extra is not installed
`max_features`	int	`4096`	TF-IDF vocabulary cap
`svd_dim`	int	`64`	SVD width in the TF-IDF fallback path
`top_examples_per_cluster`	int	`5`	How many example excerpts each cluster shows in the report
`label_top_terms`	int	`4`	How many terms make up a cluster's label

If your data already has a taxonomy, set method: declared and put cluster_id on each row. The report then speaks in your categories rather than discovered ones, which usually makes the verdict directly actionable.

train#

Settings for the optional gradian train path. These do not affect attribution.

Field	Type	Default
`backend`	`hf`, `unsloth`	`hf`
`base_model`	str	empty
`output_dir`	str	`runs/run`
`r`	int	`8`
`lora_alpha`	int	`16`
`lora_dropout`	float	`0.0`
`target_modules`	list of str or null	`null`
`learning_rate`	float	`0.0002`
`num_train_epochs`	float	`1.0`
`per_device_train_batch_size`	int	`2`
`gradient_accumulation_steps`	int	`1`
`warmup_ratio`	float	`0.03`
`lr_scheduler_type`	str	`linear`
`max_seq_length`	int	`512`
`logging_steps`	int	`1`
`save_steps`	int	`0`
`save_checkpoints`	bool	`false`
`max_grad_norm`	float	`1.0`
`weight_decay`	float	`0.0`
`load_in_4bit`	bool	`false`
`extra`	dict	`{}`

Two notes. backend: trl is not a valid value here even though --backend trl works on the command line. And load_in_4bit affects training only, since the attribution loader has no quantization path, so a 4-bit fine-tune is analyzed at full precision.

lora_dropout above zero is fine for training, and dropout is always disabled during attribution, because otherwise per-example gradients would not be reproducible.