Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation

TL;DR

This paper develops a geometric theory explaining catastrophic forgetting in Low-Rank Adaptation (LoRA) models, linking forgetting to the angle between task gradient subspaces and showing how orthogonality influences continual learning performance.

Contribution

It introduces a geometric law governing forgetting based on subspace angles, revealing rank-invariance properties and clarifying when rank impacts forgetting in LoRA models.

Findings

Forgetting correlates strongly with the minimum principal angle between task subspaces.

High subspace angles lead to rank-invariant forgetting, independent of adapter rank.

Orthogonal methods like O-LoRA perform well when natural orthogonality is high.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions. Our central finding is that forgetting is governed by a simple geometric law: $\mathcal{F} = \alpha(1 - \cos^2\theta_{\min}) + \beta$, where $\theta_{\min}$ is the minimum principal angle between task gradient subspaces. This formulation reveals an approximate rank-invariance property, at high subspace angles, forgetting becomes largely independent of the adapter rank (coefficient of variation $\approx 0.8\%$ in controlled synthetic settings; CV $\approx 10$-$19\%$ on real benchmarks, suggesting this is regime-dependent rather than absolute). We validate our theory on synthetic tasks ($r=0.994$ correlation), Split-CIFAR100 with ViT-LoRA, and sequential GLUE with RoBERTa-LoRA. Our analysis reconciles seemingly contradictory findings in the literature: we show that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high. These insights provide principled guidance for continual learning with parameter-efficient fine-tuning.