Exemplar-Free Continual Learning for State Space Models

TL;DR

This paper introduces Inf-SSM, a geometry-aware regularization method for exemplar-free continual learning in state-space models, effectively reducing forgetting and improving accuracy on sequence modeling benchmarks.

Contribution

It proposes a novel regularization technique based on Grassmannian geometry that constrains state evolution in SSMs during continual learning, with an efficient matrix equation solution.

Findings

Significant reduction in forgetting on ImageNet-R and Caltech-256 benchmarks.

Improved accuracy across sequential tasks compared to existing methods.

Efficient $ ext{O}(n^2)$ solution for regularization using SSM structure.

Abstract

State-Space Models (SSMs) excel at capturing long-range dependencies with structured recurrence, making them well-suited for sequence modeling. However, their evolving internal states pose challenges in adapting them under Continual Learning (CL). This is particularly difficult in exemplar-free settings, where the absence of prior data leaves updates to the dynamic SSM states unconstrained, resulting in catastrophic forgetting. To address this, we propose Inf-SSM, a novel and simple geometry-aware regularization method that utilizes the geometry of the infinite-dimensional Grassmannian to constrain state evolution during CL. Unlike classical continual learning methods that constrain weight updates, Inf-SSM regularizes the infinite-horizon evolution of SSMs encoded in their extended observability subspace. We show that enforcing this regularization requires solving a matrix equation known as the Sylvester equation, which typically incurs $\mathcal{O}(n^3)$ complexity. We develop a $\mathcal{O}(n^2)$ solution by exploiting the structure and properties of SSMs. This leads to an efficient regularization mechanism that can be seamlessly integrated into existing CL methods. Comprehensive experiments on challenging benchmarks, including ImageNet-R and Caltech-256, demonstrate a significant reduction in forgetting while improving accuracy across sequential tasks.