Why max_seq_length silently truncated every training completion

What this run establishes, and what it does not. A max_seq_length of 48 truncated the target of all 800 training examples and left 292 of them with zero supervised tokens. The audit caught both as critical findings before any GPU work, which is the result that matters. But the capability metric improved rather than regressed, so the attribution half of the run had nothing meaningful to explain — and Gradian said so in its own summary rather than reporting a confident diagnosis over a non-regression.

This page is published as a partial result because that is what it is.

The setup#

What the audit caught#

Three findings, two of them critical, all from the dataset and config alone — no GPU, no adapter, seconds to run.

The second one is the finding worth dwelling on. 292 of 800 examples had their prompt alone fill the window, so the target was cut off entirely and the loss was computed over nothing. More than a third of the run was doing no work at all, and nothing in the training logs would have said so — no error, no warning, a loss curve that descends perfectly normally over the examples that still had tokens left.

Gradian leads its report with these and refuses to be read past them: "2 critical issue(s) invalidate or dominate this run. Fix these before interpreting anything else."

Why the attribution half is uninformative#

The capability went up. The experiment injected a defect that ruins the fine-tune, and the metric recorded an improvement.

That is not a bug, it is the failure mode the limitations page documents directly: a log-likelihood metric can rise even under a successful sabotage, because a fine-tune mostly teaches answer format, and format is exactly what survives truncation. The model got better at looking like the training data while getting no better — and in reality worse — at the task. A contains or exact-match metric would have caught it; the log-likelihood proxy did not.

Gradian reported the consequence rather than papering over it:

• "The base model scored only -5.05 before training — this capability was largely absent before training, so the data diagnosis below explains little."
• "No individual eval item regressed, so the query gradient averages the whole eval set." With no item to contrast against, the contrastive query has nothing to subtract, and the attribution degenerates into scoring against the average.

The cluster table reflects that: all 800 examples in one cluster at -1.062 each, the flattest and weakest signal of any run in the suite.

The lesson this run actually teaches#

Run the audit first. It cost seconds, needed no GPU, and it named the real problem exactly — while the expensive influence computation on the same run produced nothing usable because the capability metric was the wrong one.

gradian diagnose --dataset train.jsonl --run runs/my-finetune --base-model meta-llama/Llama-3.2-1B

The truncation and supervised-token checks both need the tokenizer, which is why --base-model is worth passing even though no weights are loaded. See Diagnostics.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.