Self-Supervised Training Enhances Online Continual Learning

TL;DR

This paper demonstrates that self-supervised pre-training methods like MoCo-V2, Barlow Twins, and SwAV significantly improve online continual learning performance on ImageNet, especially with limited pre-training data.

Contribution

It shows that self-supervised pre-training outperforms supervised pre-training for online continual learning, providing a new approach to improve model generalization in non-stationary environments.

Findings

Self-supervised pre-training outperforms supervised pre-training on ImageNet.

The performance gains are larger with fewer pre-training samples.

Achieved a 14.95% relative increase in top-1 accuracy over prior state-of-the-art.

Abstract

In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Recently, multiple methods have been devised for incrementally learning classes on large-scale image classification tasks, such as ImageNet. State-of-the-art continual learning methods use an initial supervised pre-training phase, in which the first 10% - 50% of the classes in a dataset are used to learn representations in an offline manner before continual learning of new classes begins. We hypothesize that self-supervised pre-training could yield features that generalize better than supervised learning, especially when the number of samples used for pre-training is small. We test this hypothesis using the self-supervised MoCo-V2, Barlow Twins, and SwAV algorithms. On ImageNet, we find that these methods outperform supervised pre-training considerably for online continual learning, and the gains are larger when fewer samples are available. Our findings are consistent across three online continual learning algorithms. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.