KeepLoRA: Continual Learning with Residual Gradient Adaptation

TL;DR

KeepLoRA is a continual learning method that preserves pre-trained knowledge and adapts to new tasks by restricting updates in the residual subspace, achieving state-of-the-art results.

Contribution

It introduces a novel residual gradient adaptation approach that balances knowledge retention and plasticity in continual learning for vision-language models.

Findings

Keeps state-of-the-art performance in continual learning tasks

Effectively balances knowledge retention and plasticity

Theoretical analysis supports empirical results

Abstract

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.