Does attention-only LoRA break a fine-tune? This run says no

Targeting q_proj and v_proj while skipping the MLP was detected by the config check, but the capability improved by 3.31 nats. The audit worked; the experiment produced no regression to attribute.

Source: experiments/gpu/results/B-attention_only/

What this run establishes, and what it does not. The injected defect — LoRA adapters on q_proj and v_proj only, skipping the MLP projections entirely — was detected and named by the audit, and the training data was correctly exonerated rather than blamed. But the capability improved by 3.31 nats, so this run never produced the regression it was designed to produce. Half the experiment passed. The other half had nothing to test against.

The setup#


Base model	`unsloth/Llama-3.2-1B-Instruct`
Training data	800 clean Alpaca examples, no planted defects
Injected defect	`target_modules = ['q_proj', 'v_proj']`, MLP projections excluded
Capability	`badconfig_attention_only`, 16 eval items
Engine	`datainf`, gradient index `80582e3b6017090214ac`

The check did fire#

config.attention_only_targets, medium severity:

LoRA targets attention only: ['q_proj', 'v_proj']

Attention-only LoRA is not wrong in itself — it is a legitimate memory-saving choice and plenty of published configurations use it. The check is medium rather than high for that reason: it reports a decision worth reconsidering when a capability regresses, not an error. On this run, the decision turned out not to cost anything measurable.

But nothing regressed#

metric	base	tuned	delta	95% CI	items	verdict
`loglik`	-5.0489	-1.7381	+3.3108	[+2.7958, +3.8373]	16	improved

A 3.31 nat improvement, with the interval well clear of zero. Restricting LoRA to the attention projections did not prevent this model from learning this task.

The same caveat as the truncation run applies: loglik is a differentiable proxy, the base model scored only -5.05 beforehand, and a fine-tune that mostly teaches answer format will move that number upward almost regardless. So "improved" here means "the proxy went up", not "the model definitely got better at the underlying task". An exact-match metric might tell a different story. That experiment has not been run.

What the data verdict looked like#

cluster	n	signed influence	share	mean per example
100	800	+6.254e+04	100.0%	+78.17

Positive, and strongly so — +78.17 per example. The training data was net helpful to this capability, and Gradian reported it that way rather than manufacturing a negative cluster to accuse. That matters for the experiment's actual pass criterion, which is that the config checks name the real cause and do not blame the data. On the second half of that, this run behaves correctly.

Gradian also noted that "no individual eval item regressed, so the query gradient averages the whole eval set" — with nothing to contrast against, the contrastive query has nothing to subtract. The attribution is measuring against an average rather than against a failure.

Why this is published#

Because the honest reading is "inconclusive on the half that mattered most", and a validation suite whose public write-up quietly omitted its non-results would not be worth reading. The stronger demonstrations that a config defect is detectable and separable from a data defect are the epoch-count, learning-rate and batch-size runs, where the capability genuinely collapsed.

A better version of this experiment needs a capability that actually depends on MLP adaptation, and an exact-match metric rather than a log-likelihood proxy. It has not been designed yet.

Reproducing it#

python experiments/gpu/run_gpu_suite.py --experiment B --model unsloth/Llama-3.2-1B-Instruct

Details in GPU validation.