Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation

TL;DR

PEARL is a novel continual learning framework that dynamically allocates low-rank adaptation parameters, effectively reducing catastrophic forgetting across various vision models without rehearsal.

Contribution

It introduces a dynamic rank allocation method for LoRA components in continual learning, improving performance and resource efficiency without requiring rehearsal.

Findings

PEARL outperforms baseline methods significantly.

Effective across multiple vision architectures.

Reduces catastrophic forgetting without rehearsal.

Abstract

Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with a lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low rank adapters (LoRA) in these approaches are highly sensitive to rank selection which can lead to sub-optimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific LoRA components based on the current tasks' proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate it across three vision architectures (ResNet, Separable Convolutional Network and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin.