Spectral Adapter: Fine-Tuning in Spectral Space
Fangzhao Zhang, Mert Pilanci

TL;DR
This paper introduces Spectral Adapter, a spectral space fine-tuning method that enhances parameter-efficient fine-tuning by leveraging spectral information of pretrained weights, improving efficiency and performance.
Contribution
It proposes spectral adaptation mechanisms using SVD for PEFT, with theoretical analysis and extensive experiments demonstrating improved parameter efficiency and multi-adapter fusion.
Findings
Improved rank capacity of low-rank adapters.
Enhanced parameter efficiency and tuning performance.
Benefits in multi-adapter fusion.
Abstract
Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNumerical methods in inverse problems · Matrix Theory and Algorithms · Geophysics and Sensor Technology
