Latent Spectral Regularization for Continual Learning

TL;DR

This paper introduces CaSpeR-IL, a spectral regularizer that improves rehearsal-based continual learning by promoting better latent space partitioning, reducing class interference, and enhancing performance on benchmarks.

Contribution

It proposes a novel geometric regularizer based on Laplacian spectrum analysis that can be integrated with existing rehearsal-based CL methods.

Findings

Improves SOTA continual learning performance

Reduces class mixing in latent space

Enhances stability of predictions during learning

Abstract

While biological intelligence grows organically as new knowledge is gathered throughout life, Artificial Neural Networks forget catastrophically whenever they face a changing training data distribution. Rehearsal-based Continual Learning (CL) approaches have been established as a versatile and reliable solution to overcome this limitation; however, sudden input disruptions and memory constraints are known to alter the consistency of their predictions. We study this phenomenon by investigating the geometric characteristics of the learner's latent space and find that replayed data points of different classes increasingly mix up, interfering with classification. Hence, we propose a geometric regularizer that enforces weak requirements on the Laplacian spectrum of the latent space, promoting a partitioning behavior. Our proposal, called Continual Spectral Regularizer for Incremental Learning (CaSpeR-IL), can be easily combined with any rehearsal-based CL approach and improves the performance of SOTA methods on standard benchmarks.