C-LoRA: Continual Low-Rank Adaptation for Pre-trained Models

TL;DR

C-LoRA introduces a dynamic, scalable method for continual learning that efficiently manages parameter updates across tasks, reducing interference and improving performance in sequential learning scenarios.

Contribution

It proposes a novel learnable routing matrix in LoRA for continual learning, enabling task adaptation without separate adapters and enhancing scalability and efficiency.

Findings

Achieves state-of-the-art accuracy on benchmarks

Reduces parameter overhead compared to existing methods

Provides theoretical insights into knowledge transfer

Abstract

Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that has been extensively applied in areas such as natural language processing and computer vision. Existing LoRA fine-tuning approaches excel in static environments but struggle in dynamic learning due to reliance on multiple adapter modules, increasing overhead and complicating inference. We propose Continual Low-Rank Adaptation (C-LoRA), a novel extension of LoRA for continual learning. C-LoRA uses a learnable routing matrix to dynamically manage parameter updates across tasks, ensuring efficient reuse of learned subspaces while enforcing orthogonality to minimize interference and forgetting. Unlike existing approaches that require separate adapters for each task, C-LoRA enables a integrated approach for task adaptation, achieving both scalability and parameter efficiency in sequential learning scenarios. C-LoRA achieves state-of-the-art accuracy and parameter efficiency on benchmarks while providing theoretical insights into its routing matrix's role in retaining and transferring knowledge, establishing a scalable framework for continual learning.