PLAN: Proactive Low-Rank Allocation for Continual Learning

TL;DR

PLAN introduces a proactive low-rank allocation framework that enhances continual learning by efficiently managing task-specific subspaces, reducing interference, and outperforming existing methods on standard benchmarks.

Contribution

The paper presents a novel proactive low-rank allocation method extending LoRA for interference-aware continual learning with foundation models.

Findings

PLAN outperforms existing continual learning methods on standard benchmarks.

The approach effectively reduces interference between tasks.

Empirical results demonstrate state-of-the-art performance in continual learning scenarios.

Abstract

Continual learning (CL) requires models to continuously adapt to new tasks without forgetting past knowledge. In this work, we propose \underline{P}roactive \underline{L}ow-rank \underline{A}llocatio\underline{N} (PLAN), a framework that extends Low-Rank Adaptation (LoRA) to enable efficient and interference-aware fine-tuning of large pre-trained models in CL settings. PLAN proactively manages the allocation of task-specific subspaces by introducing orthogonal basis vectors for each task and optimizing them through a perturbation-based strategy that minimizes conflicts with previously learned parameters. Furthermore, PLAN incorporates a novel selection mechanism that identifies and assigns basis vectors with minimal sensitivity to interference, reducing the risk of degrading past knowledge while maintaining efficient adaptation to new tasks. Empirical results on standard CL benchmarks demonstrate that PLAN consistently outperforms existing methods, establishing a new state-of-the-art for continual learning with foundation models.