Exemplar-free Continual Representation Learning via Learnable Drift Compensation

TL;DR

This paper introduces Learnable Drift Compensation (LDC), a novel method to mitigate semantic drift in exemplar-free continual learning, enabling effective semi-supervised and supervised continual representation learning with state-of-the-art results.

Contribution

The paper proposes LDC, a simple and effective drift correction method applicable to any backbone, enhancing exemplar-free continual learning, including semi-supervised scenarios.

Findings

LDC effectively mitigates semantic drift in continual learning.

State-of-the-art performance achieved in supervised and semi-supervised settings.

LDC is easy to integrate with existing continual learning approaches.

Abstract

Exemplar-free class-incremental learning using a backbone trained from scratch and starting from a small first task presents a significant challenge for continual representation learning. Prototype-based approaches, when continually updated, face the critical issue of semantic drift due to which the old class prototypes drift to different positions in the new feature space. Through an analysis of prototype-based continual learning, we show that forgetting is not due to diminished discriminative power of the feature extractor, and can potentially be corrected by drift compensation. To address this, we propose Learnable Drift Compensation (LDC), which can effectively mitigate drift in any moving backbone, whether supervised or unsupervised. LDC is fast and straightforward to integrate on top of existing continual learning approaches. Furthermore, we showcase how LDC can be applied in combination with self-supervised CL methods, resulting in the first exemplar-free semi-supervised continual learning approach. We achieve state-of-the-art performance in both supervised and semi-supervised settings across multiple datasets. Code is available at \url{https://github.com/alviur/ldc}.