SORSA: Singular Values and Orthonormal Regularized Singular Vectors Adaptation of Large Language Models

TL;DR

SORSA introduces a novel PEFT method using SVD-based adapters with orthonormal regularization, achieving faster convergence and superior performance in large language model fine-tuning.

Contribution

The paper presents SORSA, a new PEFT approach that leverages SVD and orthonormal regularization for efficient and effective large language model adaptation.

Findings

SORSA outperforms LoRA and full fine-tuning in convergence speed.

On GSM-8K, SORSA achieves 56.03% accuracy, higher than LoRA and full FT.

SORSA adapters can be merged during inference, eliminating latency.

Abstract

In this paper, we propose Singular Values and Orthonormal Regularized Singular Vectors Adaptation, or SORSA, a novel parameter efficient fine-tuning (PEFT) method. Each SORSA adapter consists of two main parts: trainable principal singular weights $W_p = U_p \text{diag}(S_p) V^\top_p$, and frozen residual weights $W_r = U_r \text{diag}(S_r) V^\top_r$. These parts are initialized by performing singular value decomposition (SVD) on pre-trained weights. Moreover, we implement and analyze an orthonormal regularizer, which we prove could decrease the condition number of $W_p$ and make the optimization more efficient. SORSA adapters could be merged during inference, thus eliminating any inference latency. We also introduce a method to analyze the variation of the parameters by performing SVD and discuss and analyze SORSA's superiority in minimizing the alteration in the SVD aspect. After all, SORSA shows a faster convergence than LoRA and PiSSA in our experiments. On the GSM-8K benchmark, Llama 2 7B adapted using SORSA achieved 56.03\% accuracy, surpassing LoRA (42.30\%) and Full FT (49.05\%). We conclude that SORSA offers a new perspective on parameter-efficient fine-tuning, demonstrating remarkable performance.