LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning

TL;DR

LARGO is a novel low-rank gradient projection method that enhances robustness and efficiency in parameter-efficient fine-tuning of large models, especially under domain shifts.

Contribution

It introduces a dynamic gradient regulation technique with SVD-based initialization, improving robustness and reducing computational costs in PEFT.

Findings

Achieves state-of-the-art robustness under domain shifts.

Maintains high performance with lower computational overhead.

Effectively preserves layer independence during fine-tuning.

Abstract

The advent of parameter-efficient fine-tuning methods has significantly reduced the computational burden of adapting large-scale pretrained models to diverse downstream tasks. However, existing approaches often struggle to achieve robust performance under domain shifts while maintaining computational efficiency. To address this challenge, we propose Low-rAnk Regulated Gradient Projection (LARGO) algorithm that integrates dynamic constraints into low-rank adaptation methods. Specifically, LARGO incorporates parallel trainable gradient projections to dynamically regulate layer-wise updates, retaining the Out-Of-Distribution robustness of pretrained model while preserving inter-layer independence. Additionally, it ensures computational efficiency by mitigating the influence of gradient dependencies across layers during weight updates. Besides, through leveraging singular value decomposition of pretrained weights for structured initialization, we incorporate an SVD-based initialization strategy that minimizing deviation from pretrained knowledge. Through extensive experiments on diverse benchmarks, LARGO achieves state-of-the-art performance across in-domain and out-of-distribution scenarios, demonstrating improved robustness under domain shifts with significantly lower computational overhead compared to existing PEFT methods. The source code will be released soon.