Continual Learning with Low Rank Adaptation

TL;DR

This paper introduces CoLoR, a low-rank adaptation method for continual learning with transformers, achieving state-of-the-art results while maintaining parameter efficiency across various domain-incremental benchmarks.

Contribution

It demonstrates that Low Rank Adaptation (LoRA) can be effectively applied to continual learning, outperforming prompt tuning methods in both performance and efficiency.

Findings

CoLoR achieves state-of-the-art results on domain-incremental benchmarks.

LoRA-based continual learning is as parameter-efficient as prompt tuning.

The approach effectively retains performance on previous data while adapting to new data.

Abstract

Recent work using pretrained transformers has shown impressive performance when fine-tuned with data from the downstream problem of interest. However, they struggle to retain that performance when the data characteristics changes. In this paper, we focus on continual learning, where a pre-trained transformer is updated to perform well on new data, while retaining its performance on data it was previously trained on. Earlier works have tackled this primarily through methods inspired from prompt tuning. We question this choice, and investigate the applicability of Low Rank Adaptation (LoRA) to continual learning. On a range of domain-incremental learning benchmarks, our LoRA-based solution, CoLoR, yields state-of-the-art performance, while still being as parameter efficient as the prompt tuning based methods.