CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation

TL;DR

CLoRA introduces a parameter-efficient method for continual learning that reduces computational costs and resource requirements, enabling effective learning across tasks without retraining entire models.

Contribution

The paper proposes CLoRA, a novel approach using Low-Rank Adaptation for resource-efficient continual learning in neural networks.

Findings

Achieves comparable or better performance than baseline methods.

Significantly reduces hardware and computational requirements.

Effective in resource-constrained environments.

Abstract

In the past, continual learning (CL) was mostly concerned with the problem of catastrophic forgetting in neural networks, that arises when incrementally learning a sequence of tasks. Current CL methods function within the confines of limited data access, without any restrictions imposed on computational resources. However, in real-world scenarios, the latter takes precedence as deployed systems are often computationally constrained. A major drawback of most CL methods is the need to retrain the entire model for each new task. The computational demands of retraining large models can be prohibitive, limiting the applicability of CL in environments with limited resources. Through CLoRA, we explore the applicability of Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method for class-incremental semantic segmentation. CLoRA leverages a small set of parameters of the model and uses the same set for learning across all tasks. Results demonstrate the efficacy of CLoRA, achieving performance on par with and exceeding the baseline methods. We further evaluate CLoRA using NetScore, underscoring the need to factor in resource efficiency and evaluate CL methods beyond task performance. CLoRA significantly reduces the hardware requirements for training, making it well-suited for CL in resource-constrained environments after deployment.