Norm-Bounded Low-Rank Adaptation

TL;DR

This paper introduces NB-LoRA, a novel low-rank adaptation method with explicit norm bounds, enhancing parameter-efficient fine-tuning robustness and performance in language and vision tasks.

Contribution

It proposes a new norm-bounded low-rank adaptation parameterization that guarantees norm constraints and improves robustness over existing methods.

Findings

Matches or surpasses performance of existing LoRA methods in language tasks.

Enhances robustness to hyper-parameters like rank, learning rate, and epochs.

Reduces catastrophic forgetting in vision fine-tuning.

Abstract

In this work, we propose norm-bounded low-rank adaptation (NB-LoRA) for parameter-efficient fine tuning. NB-LoRA is a novel parameterization of low-rank weight adaptations that admits explicit bounds on each singular value of the adaptation matrix, which can thereby satisfy any prescribed unitarily invariant norm bound, including the Schatten norms (e.g., nuclear, Frobenius, spectral norm). The proposed parameterization is unconstrained, smooth, and complete, i.e. it covers all matrices satisfying the prescribed rank and singular-value bounds. Natural language generation experiments show that NB-LoRA matches or surpasses performance of competing LoRA methods, while exhibiting stronger hyper-parameter robustness. Vision fine-tuning experiments show that NB-LoRA can avoid model catastrophic forgetting without minor cost on adaptation performance, and compared to existing approaches it is substantially more robust to a hyper-parameters such as including adaptation rank, learning rate and number of training epochs.