Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning

TL;DR

This paper introduces an adversarial perturbation method to estimate and compensate for feature drift in exemplar-free continual learning, significantly improving performance on standard benchmarks and challenging settings.

Contribution

It proposes a novel adversarial approach to track and correct feature drift in exemplar-free continual learning, enhancing model stability without storing exemplars.

Findings

Outperforms existing methods on standard benchmarks

Better tracks prototype movement in embedding space

Effective in challenging small initial task settings

Abstract

Continual learning methods are known to suffer from catastrophic forgetting, a phenomenon that is particularly hard to counter for methods that do not store exemplars of previous tasks. Therefore, to reduce potential drift in the feature extractor, existing exemplar-free methods are typically evaluated in settings where the first task is significantly larger than subsequent tasks. Their performance drops drastically in more challenging settings starting with a smaller first task. To address this problem of feature drift estimation for exemplar-free methods, we propose to adversarially perturb the current samples such that their embeddings are close to the old class prototypes in the old model embedding space. We then estimate the drift in the embedding space from the old to the new model using the perturbed images and compensate the prototypes accordingly. We exploit the fact that adversarial samples are transferable from the old to the new feature space in a continual learning setting. The generation of these images is simple and computationally cheap. We demonstrate in our experiments that the proposed approach better tracks the movement of prototypes in embedding space and outperforms existing methods on several standard continual learning benchmarks as well as on fine-grained datasets. Code is available at https://github.com/dipamgoswami/ADC.