Source: https://gradian.dev/docs/diagnostics

# Diagnostics

Not every broken fine-tune is a data problem. Truncated completions, a loss mask covering the prompt,
missing EOS, train and eval contamination, a learning rate set for full fine-tuning, twelve epochs
over 400 examples. None of these crash, none show up as a training error, and no amount of influence
math explains them better than simply checking.

```bash
gradian diagnose --dataset train.jsonl --run runs/my-finetune --base-model meta-llama/Llama-3.2-1B
```

Seconds, no GPU, no model weights. Run this before anything expensive.

Each finding carries a severity, the evidence that triggered it, the specific examples involved where
that applies, and a concrete fix. The attribution report merges these in, so when the cluster that hurt
you turns out to be the truncated one, the report says so.

## What each input unlocks

| You pass | You get |
|---|---|
| `--dataset` | The 16 `data.*` checks, minus the ones needing a tokenizer |
| `--base-model` | Tokenizer-dependent checks: truncation, EOS, loss mask coverage |
| `--run` | The 17 `config.*` and 8 `dynamics.*` checks |
| `--eval-dataset` | Train and eval contamination |

Passing only `--dataset` is useful, but you are leaving most of the value on the table. The config and
dynamics checks are two thirds of the list.

## The 41 checks

Listed exactly as `gradian checks` prints them.

### Config, 17 checks

Read from the trainer config in your run directory.

| Check | What it catches |
|---|---|
| `config.attention_only_targets` | target_modules covers attention only |
| `config.base_model_mismatch` | Adapter trained on a different base model |
| `config.capacity_vs_data` | Adapter capacity large relative to the data |
| `config.fp16_precision` | fp16 rather than bf16 |
| `config.grad_clip_disabled` | Gradient clipping disabled |
| `config.high_lora_dropout` | High LoRA dropout on a small dataset |
| `config.lr_too_high` | Learning rate too high for LoRA |
| `config.lr_too_low` | Learning rate too low to learn |
| `config.max_seq_length_vs_data` | max_seq_length shorter than the data needs |
| `config.no_eval_set` | No eval set configured |
| `config.no_warmup` | No warmup with a high learning rate |
| `config.overtraining` | Too many epochs for the dataset size |
| `config.packing_with_completion_loss` | Packing combined with completion-only loss |
| `config.pad_equals_eos` | pad_token is the eos_token |
| `config.seed_unset` | No seed recorded |
| `config.small_effective_batch` | Effective batch size of 1-2 with a high LR |
| `config.undertrained` | Too few optimizer steps |

### Data, 16 checks

Read from the dataset itself, several of them needing the tokenizer.

| Check | What it catches |
|---|---|
| `data.completion_truncated` | Targets cut off by max_seq_length |
| `data.conflicting_answers` | Same question, disagreeing answers |
| `data.duplicates` | Exact duplicate examples |
| `data.empty_or_degenerate_targets` | Empty or trivially short targets |
| `data.format_inconsistency` | Mixed target formats |
| `data.length_outliers` | Extremely long examples |
| `data.loss_mask_coverage` | Fraction of tokens the loss is computed on |
| `data.missing_eos` | Targets that do not end with EOS |
| `data.near_duplicates` | Near-duplicate examples |
| `data.no_chat_template` | Chat data but no chat template |
| `data.no_supervised_tokens` | Examples that train on nothing |
| `data.prompt_equals_target` | Target identical to the prompt |
| `data.template_prefix_mismatch` | Chat template is not prefix-stable |
| `data.tiny_dataset` | Dataset too small for reliable attribution |
| `data.train_eval_contamination` | Eval items that also appear in training |
| `data.truncated` | Examples truncated at all |

### Dynamics, 8 checks

Read from the trainer's logged metrics.

| Check | What it catches |
|---|---|
| `dynamics.always_clipping` | Gradient norm pinned at the clip threshold |
| `dynamics.grad_norm_explosion` | Gradient-norm blowup |
| `dynamics.loss_plateau` | Loss barely moved |
| `dynamics.loss_spike` | Sudden loss spikes |
| `dynamics.lr_schedule_incomplete` | LR schedule did not decay |
| `dynamics.nan_loss` | NaN or inf loss |
| `dynamics.overfitting` | Train loss falling while eval loss rises |
| `dynamics.too_few_steps` | Barely any logged steps |

## The four worth knowing before you train

These four account for a disproportionate share of fine-tunes that fail without any error message.

**`data.completion_truncated`.** Your `max_seq_length` cuts the answer off mid-sentence. The model
learns to produce answers that stop abruptly, and every metric that checks the whole answer collapses.
This is the check most often behind a regression that looks like a data-quality problem.

**`data.no_supervised_tokens`.** Some examples train on nothing at all, usually because the loss mask
and the chat template disagree. Those examples contribute zero gradient, so they are invisible to
attribution and to your loss curve alike.

**`data.missing_eos`.** Without EOS on the target, the model never learns to stop, so it runs to the
generation cap and fails exact-match on answers it actually knew.

**`config.packing_with_completion_loss`.** Packing concatenates examples, completion-only masking
assumes they are separate. Together they mask the wrong tokens.

## Extending the checks

A new check is one function plus a registry entry. The project requires two fixtures for each: one
that trips the check and one that does not. There is also a false-positive guard asserting that a
healthy run produces no high-severity findings, so a check that fires too eagerly fails the suite.