Spectral-Aware Low-Rank Adaptation for Speaker Verification

TL;DR

This paper introduces a spectral-aware low-rank adaptation method for speaker verification that leverages the principal singular vectors of pre-trained models to improve fine-tuning performance.

Contribution

It proposes a novel spectral adaptation strategy that incorporates spectral information into PEFT, focusing on the additive adjustment of top singular vectors for better speaker verification.

Findings

Enhanced speaker verification accuracy on VoxCeleb1 and CN-Celeb1 datasets.

Spectral adaptation outperforms traditional PEFT methods.

Code released for reproducibility and further research.

Abstract

Previous research has shown that the principal singular vectors of a pre-trained model's weight matrices capture critical knowledge. In contrast, those associated with small singular values may contain noise or less reliable information. As a result, the LoRA-based parameter-efficient fine-tuning (PEFT) approach, which does not constrain the use of the spectral space, may not be effective for tasks that demand high representation capacity. In this study, we enhance existing PEFT techniques by incorporating the spectral information of pre-trained weight matrices into the fine-tuning process. We investigate spectral adaptation strategies with a particular focus on the additive adjustment of top singular vectors. This is accomplished by applying singular value decomposition (SVD) to the pre-trained weight matrices and restricting the fine-tuning within the top spectral space. Extensive speaker verification experiments on VoxCeleb1 and CN-Celeb1 demonstrate enhanced tuning performance with the proposed approach. Code is released at https://github.com/lizhepolyu/SpectralFT.