Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

TL;DR

This paper identifies spectral imbalance in low-rank adaptations as a cause of forgetting in continual learning and proposes a constrained optimization method to balance components, reducing forgetting and improving performance.

Contribution

It introduces a novel spectral balancing approach for low-rank continual adaptation, addressing imbalance issues to mitigate forgetting.

Findings

Reduces backward and forward forgetting in experiments.

Outperforms baseline continual learning methods.

Balances spectral components effectively.

Abstract

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.