SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning

TL;DR

SD-LoRA introduces a scalable, decoupled low-rank adaptation method for class incremental learning that improves stability and plasticity without requiring rehearsal, enabling efficient continual learning with foundation models.

Contribution

The paper proposes SD-LoRA, a novel method that decouples learning of LoRA components, enhancing scalability and parameter efficiency in continual learning without rehearsal.

Findings

SD-LoRA achieves superior stability-plasticity trade-off.

It converges to a low-loss region across tasks.

Experimental results validate its effectiveness across benchmarks.

Abstract

Continual Learning (CL) with foundation models has recently emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks. However, existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks, which poses significant scalability challenges as the number of tasks grows. To address these limitations, we propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal. Our empirical and theoretical analysis reveals that SD-LoRA tends to follow a low-loss trajectory and converges to an overlapping low-loss region for all learned tasks, resulting in an excellent stability-plasticity trade-off. Building upon these insights, we introduce two variants of SD-LoRA with further improved parameter efficiency. All parameters of SD-LoRAs can be end-to-end optimized for CL objectives. Meanwhile, they support efficient inference by allowing direct evaluation with the finally trained model, obviating the need for component selection. Extensive experiments across multiple CL benchmarks and foundation models consistently validate the effectiveness of SD-LoRA. The code is available at https://github.com/WuYichen-97/SD-Lora-CL.