Shared LoRA Subspaces for almost Strict Continual Learning

TL;DR

This paper introduces Share, a parameter-efficient continual learning method that uses a shared low-rank subspace to adapt large models across multiple tasks without catastrophic forgetting, significantly reducing parameters and memory.

Contribution

Share is a novel approach that constructs and updates a single shared subspace for continual learning, enabling scalable, data-free knowledge transfer across diverse modalities.

Findings

Achieves up to 100x parameter reduction compared to traditional methods.

Provides 281x memory savings over standard LoRA techniques.

Maintains performance comparable to joint training across multiple tasks.

Abstract

Adapting large pretrained models to new tasks efficiently and continually is crucial for real-world deployment but remains challenging due to catastrophic forgetting and the high cost of retraining. While parameter-efficient tuning methods like low rank adaptation (LoRA) reduce computational demands, they lack mechanisms for strict continual learning and knowledge integration, without relying on data replay, or multiple adapters. We propose Share, a novel approach to parameter efficient continual finetuning that learns and dynamically updates a single, shared low-rank subspace, enabling seamless adaptation across multiple tasks and modalities. Share constructs a foundational subspace that extracts core knowledge from past tasks and incrementally integrates new information by identifying essential subspace directions. Knowledge from each new task is incorporated into this evolving subspace, facilitating forward knowledge transfer, while minimizing catastrophic interference. This approach achieves up to 100x parameter reduction and 281x memory savings over traditional LoRA methods, maintaining performance comparable to jointly trained models. A single Share model can replace hundreds of task-specific LoRA adapters, supporting scalable, asynchronous continual learning. Experiments across image classification, natural language understanding, 3D pose estimation, and text-to-image generation validate its effectiveness, making Share a practical and scalable solution for lifelong learning in large-scale AI systems.