Regularizing Subspace Redundancy of Low-Rank Adaptation

TL;DR

ReSoRA is a novel regularization method that reduces redundancy in low-rank adaptation matrices, improving parameter efficiency and transfer learning performance across various vision-language and visual classification tasks.

Contribution

It introduces a theoretically grounded approach to explicitly model and regularize subspace redundancy in LoRA, enhancing its effectiveness and generalization.

Findings

Consistently improves state-of-the-art PETL methods

Enhances performance across diverse datasets and architectures

No additional inference costs introduced

Abstract

Low-Rank Adaptation (LoRA) and its variants have delivered strong capability in Parameter-Efficient Transfer Learning (PETL) by minimizing trainable parameters and benefiting from reparameterization. However, their projection matrices remain unrestricted during training, causing high representation redundancy and diminishing the effectiveness of feature adaptation in the resulting subspaces. While existing methods mitigate this by manually adjusting the rank or implicitly applying channel-wise masks, they lack flexibility and generalize poorly across various datasets and architectures. Hence, we propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation. Specifically, it theoretically decomposes the low-rank submatrices into multiple equivalent subspaces and systematically applies de-redundancy constraints to the feature distributions across different projections. Extensive experiments validate that our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks. Besides, as a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs. Code is publicly available at: https://github.com/Lucenova/ReSoRA.