Gradian

Diagnostics

The 41 deterministic checks Gradian runs over your dataset, config and trainer logs, with no GPU and no attribution involved.

View .md

Not every broken fine-tune is a data problem. Truncated completions, a loss mask covering the prompt, missing EOS, train and eval contamination, a learning rate set for full fine-tuning, twelve epochs over 400 examples. None of these crash, none show up as a training error, and no amount of influence math explains them better than simply checking.

gradian diagnose --dataset train.jsonl --run runs/my-finetune --base-model meta-llama/Llama-3.2-1B

Seconds, no GPU, no model weights. Run this before anything expensive.

Each finding carries a severity, the evidence that triggered it, the specific examples involved where that applies, and a concrete fix. The attribution report merges these in, so when the cluster that hurt you turns out to be the truncated one, the report says so.

What each input unlocks#

You pass You get
--dataset The 16 data.* checks, minus the ones needing a tokenizer
--base-model Tokenizer-dependent checks: truncation, EOS, loss mask coverage
--run The 17 config.* and 8 dynamics.* checks
--eval-dataset Train and eval contamination

Passing only --dataset is useful, but you are leaving most of the value on the table. The config and dynamics checks are two thirds of the list.

The 41 checks#

Listed exactly as gradian checks prints them.

Config, 17 checks#

Read from the trainer config in your run directory.

Check What it catches
config.attention_only_targets target_modules covers attention only
config.base_model_mismatch Adapter trained on a different base model
config.capacity_vs_data Adapter capacity large relative to the data
config.fp16_precision fp16 rather than bf16
config.grad_clip_disabled Gradient clipping disabled
config.high_lora_dropout High LoRA dropout on a small dataset
config.lr_too_high Learning rate too high for LoRA
config.lr_too_low Learning rate too low to learn
config.max_seq_length_vs_data max_seq_length shorter than the data needs
config.no_eval_set No eval set configured
config.no_warmup No warmup with a high learning rate
config.overtraining Too many epochs for the dataset size
config.packing_with_completion_loss Packing combined with completion-only loss
config.pad_equals_eos pad_token is the eos_token
config.seed_unset No seed recorded
config.small_effective_batch Effective batch size of 1-2 with a high LR
config.undertrained Too few optimizer steps

Data, 16 checks#

Read from the dataset itself, several of them needing the tokenizer.

Check What it catches
data.completion_truncated Targets cut off by max_seq_length
data.conflicting_answers Same question, disagreeing answers
data.duplicates Exact duplicate examples
data.empty_or_degenerate_targets Empty or trivially short targets
data.format_inconsistency Mixed target formats
data.length_outliers Extremely long examples
data.loss_mask_coverage Fraction of tokens the loss is computed on
data.missing_eos Targets that do not end with EOS
data.near_duplicates Near-duplicate examples
data.no_chat_template Chat data but no chat template
data.no_supervised_tokens Examples that train on nothing
data.prompt_equals_target Target identical to the prompt
data.template_prefix_mismatch Chat template is not prefix-stable
data.tiny_dataset Dataset too small for reliable attribution
data.train_eval_contamination Eval items that also appear in training
data.truncated Examples truncated at all

Dynamics, 8 checks#

Read from the trainer's logged metrics.

Check What it catches
dynamics.always_clipping Gradient norm pinned at the clip threshold
dynamics.grad_norm_explosion Gradient-norm blowup
dynamics.loss_plateau Loss barely moved
dynamics.loss_spike Sudden loss spikes
dynamics.lr_schedule_incomplete LR schedule did not decay
dynamics.nan_loss NaN or inf loss
dynamics.overfitting Train loss falling while eval loss rises
dynamics.too_few_steps Barely any logged steps

The four worth knowing before you train#

These four account for a disproportionate share of fine-tunes that fail without any error message.

data.completion_truncated. Your max_seq_length cuts the answer off mid-sentence. The model learns to produce answers that stop abruptly, and every metric that checks the whole answer collapses. This is the check most often behind a regression that looks like a data-quality problem.

data.no_supervised_tokens. Some examples train on nothing at all, usually because the loss mask and the chat template disagree. Those examples contribute zero gradient, so they are invisible to attribution and to your loss curve alike.

data.missing_eos. Without EOS on the target, the model never learns to stop, so it runs to the generation cap and fails exact-match on answers it actually knew.

config.packing_with_completion_loss. Packing concatenates examples, completion-only masking assumes they are separate. Together they mask the wrong tokens.

Extending the checks#

A new check is one function plus a registry entry. The project requires two fixtures for each: one that trips the check and one that does not. There is also a false-positive guard asserting that a healthy run produces no high-severity findings, so a check that fires too eagerly fails the suite.