S2FT: Parameter-Efficient Fine-Tuning in Sparse Spectrum Domain

TL;DR

This paper introduces S2FT, a parameter-efficient fine-tuning method that transforms weight changes into a sparse spectrum domain using an invertible transformation, enabling effective adaptation with minimal parameters.

Contribution

Proposes a novel invertible transformation approach for PEFT that models non-sparse spectral weight changes by transforming them into a sparse spectrum domain.

Findings

S2FT achieves superior performance with only 0.08% of training parameters.

The method effectively models weight changes with a non-sparse, power-uniform spectrum.

Extensive experiments validate the efficiency and effectiveness of S2FT.

Abstract

Parameter Efficient Fine-Tuning (PEFT) is a key technique for adapting a large pretrained model to downstream tasks by fine-tuning only a small number of parameters. Recent methods based on Fourier transforms have further reduced the fine-tuned parameters scale by only fine-tuning a few spectral coefficients. Its basic assumption is that the weight change \delta W is a spatial-domain matrix with a sparse spectrum. However, in this paper, we observe that the spectrum of weight change is not sparse, but instead distributed like power-uniform. This fact implies that fine-tuning only a few spectral coefficients is insufficient to accurately model the weight change with uniform spectrum. To address this issue, we propose to seek an invertible transformation that can transform a latent spatial-domain matrix with sparse spectrum to the weight change, and then perform PEFT on such sparse spectrum domain with few spectral coefficients, called S2FT. To seek such transformation, we first pre-estimate a coarse weight change as a prior. Then, inspired by that sparse spectrum often correspond to locally smooth spatial structures, we regard this transformation as a row and column rearrangement operation on the pre-estimated weight change that smooth spatial structures while keep the structure information of neurons. Finally, we propose to solve the rearrangement search problem in a simple nearest neighbor search manner, thereby obtaining the invertible transformation. Extensive results show our S2FT achieves superior performance by only using 0.08% training parameters.