Cross-scale adapter transfer: a 32B model's LoRA onto its 4B sibling

The short version

This is a controlled model-training experiment from August 2026: whether a 32B image model's LoRA adapter could be projected onto its 4B distilled sibling by linear algebra alone, to render cheap previews without pausing training. The answer, reached with real controls, was no. It proves the discipline of a falsifiable test: I built the null distributions and shuffled-block controls that make a negative result mean something, then separated a weak real signal from the noise it hides in. The transferable skill is research rigor: a path closed with evidence, not a hunch, and the habit of reporting a null distribution instead of a single draw.

The method

I built per-layer SVD caches of both base models, decomposing each weight matrix into its principal directions. Blocks were paired by relative depth, on the assumption that a distilled student's depth tracks its teacher's; the shuffled-block control below is partly a test of that assumption. The projection mapped source singular direction i to target singular direction i. A factored implementation was required, because the dense path was intractable per module.

The obvious gating metric does not work here. With orthonormal bases, a minimum-norm map sends any r orthonormal vectors onto any other r, so subspace-similarity gating has zero discriminative power across different widths. The proxy I used instead was a source-side capture fraction: the share of the source adapter's energy inside its own top-k singular directions. It bounds truncation loss but says nothing about cross-model geometry, which is why the transfer question could only be settled by activation-space controls.

The result

The shortcut failed. Weight-space transfer carried no actionable information, and depth-paired spectral transfer was statistically indistinguishable from a shuffled-block correspondence.

The one positive signal was in activation space, in the early double blocks. Blocks 0, 2, and 4 sat above every draw of a 12-draw null, at +5.8, +5.0, and +2.6 standard deviations. The magnitudes are coarse at that sample size; the ordering is the finding. The deep double blocks did not clear their own noise baseline, and all 20 single-block comparisons were empty. The early-block result held across two captures and three null distributions.

The method rule I took from it: one derangement is a single null draw, not a null distribution, so report the distribution. The report also went through an adversarial review pass, which is part of the record.

What the result settles, and what it does not

The transfer question has its answer. No configuration produced a usable preview, and the failure is geometric, not a scaling error. The cheap honest preview the lab runs instead is a small klein adapter trained on the same corpus, which forecasts corpus health before a long 32B run; it does not preview a specific checkpoint, because nothing that samples the 32B function can be made free.

The activation-space signal in the early blocks is the one thread this left hanging. It is a measurement, not a method: it says a correspondence exists there, not that an adapter can be routed through it. The compute ran as scheduled jobs on the lab's fleet, with load-aware placement and clean cancellation, infrastructure that outlived the experiment.

The QLoRA 32B project is the training work this sits beside, and the image-generation rounds are where the need for a cheap preview came from.