Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation

TL;DR

This paper reveals that pretrained models establish a stable spectral basis that is shared across tasks, and demonstrates that freezing these spectral directions with minimal trainable parameters can effectively adapt models to new tasks.

Contribution

The study uncovers the spectral stability of pretrained models across tasks and introduces a parameter-efficient adaptation method based on spectral coefficients.

Findings

Leading singular vectors are highly stable during finetuning.

Pretraining on larger datasets enhances spectral stability under shifts.

Freezing spectral vectors and tuning coefficients performs well on downstream tasks.

Abstract

Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space. Prior work has focused on characterizing this optimization subspace, but largely ignored the complementary question: why do certain directions remain unexplored during finetuning? Are these stable directions irrelevant to downstream tasks, or do they already encode task-relevant structure that requires no further adjustment? Answering this question is central to understanding how pretrained knowledge transfers. Through systematic spectral analysis across vision and language models, we show that the leading singular vectors of pretrained weight matrices remain highly stable under finetuning and are shared across unrelated downstream tasks, revealing that pretraining establishes a reusable spectral coordinate system. Models pretrained on larger datasets exhibit greater spectral stability under distribution shift or task change, directly linking pretraining scale to geometric transferability. Motivated by these findings, we propose a parameter-efficient method that freezes pretrained singular vectors and optimizes only leading spectral coefficients, achieving competitive performance on GLUE with 0.2% trainable parameters. Our results reveal that the stable directions encode transferable structure rather than irrelevant noise: successful pretraining discovers spectral bases that downstream tasks inherit and operate within.