Continuous Subspace Optimization for Continual Learning

TL;DR

This paper introduces CoSO, a novel continual learning method that dynamically optimizes in sequential subspaces to better preserve prior knowledge and improve performance on long task sequences.

Contribution

CoSO proposes a dynamic subspace optimization approach that updates task-specific directions and orthogonalizes to previous tasks, enhancing continual learning performance.

Findings

Outperforms state-of-the-art methods on multiple datasets.

Effectively mitigates catastrophic forgetting in long task sequences.

Maintains high accuracy across diverse continual learning scenarios.

Abstract

Continual learning aims to learn multiple tasks sequentially while preserving prior knowledge, but faces the challenge of catastrophic forgetting when adapting to new tasks. Recently, approaches leveraging pre-trained models have gained increasing popularity in mitigating this issue, due to the strong generalization ability of foundation models. To adjust pre-trained models for new tasks, existing methods usually employ low-rank adaptation, which restricts parameter updates to a fixed low-rank subspace. However, constraining the optimization space inherently compromises the model's learning capacity, resulting in inferior performance. To address this limitation, we propose Continuous Subspace Optimization for Continual Learning (CoSO) to fine-tune the model in a series of subspaces rather than a single one. These sequential subspaces are dynamically determined through the singular value decomposition of the gradients. CoSO updates the model by projecting gradients onto these subspaces, ensuring memory-efficient optimization. To mitigate forgetting, the optimization subspace of each task is constrained to be orthogonal to the historical task subspace. During task learning, CoSO maintains a task-specific component that captures the critical update directions for the current task. Upon completing a task, this component is used to update the historical task subspace, laying the groundwork for subsequent learning. Extensive experiments on multiple datasets demonstrate that CoSO significantly outperforms state-of-the-art methods, especially in challenging scenarios with long task sequences.