SVFit: Parameter-Efficient Fine-Tuning of Large Pre-Trained Models Using Singular Values

TL;DR

SVFit introduces a parameter-efficient fine-tuning method that uses singular value decomposition to initialize low-rank matrices, significantly reducing parameters and improving adaptation in large pre-trained models.

Contribution

SVFit proposes a novel SVD-based initialization for low-rank matrices in PEFT, enhancing efficiency and performance over existing methods like LoRA.

Findings

Outperforms LoRA in various tasks

Requires 16 times fewer trainable parameters

Achieves rapid domain adaptation

Abstract

Large pre-trained models (LPMs) have demonstrated exceptional performance in diverse natural language processing and computer vision tasks. However, fully fine-tuning these models poses substantial memory challenges, particularly in resource-constrained environments. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, mitigate this issue by adjusting only a small subset of parameters. Nevertheless, these methods typically employ random initialization for low-rank matrices, which can lead to inefficiencies in gradient descent and diminished generalizability due to suboptimal starting points. To address these limitations, we propose SVFit, a novel PEFT approach that leverages singular value decomposition (SVD) to initialize low-rank matrices using critical singular values as trainable parameters. Specifically, SVFit performs SVD on the pre-trained weight matrix to obtain the best rank-r approximation matrix, emphasizing the most critical singular values that capture over 99% of the matrix's information. These top-r singular values are then used as trainable parameters to scale the fundamental subspaces of the matrix, facilitating rapid domain adaptation. Extensive experiments across various pre-trained models in natural language understanding, text-to-image generation, and image classification tasks reveal that SVFit outperforms LoRA while requiring 16 times fewer trainable parameters.