Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learning

TL;DR

This paper introduces a class incremental learning method that uses self-supervised pre-training and prototype learning to effectively mitigate catastrophic forgetting without relying on old class exemplars, demonstrating significant performance improvements.

Contribution

It proposes a novel two-stage framework with a fixed self-supervised encoder and an incrementally updated prototype classifier, addressing representation drift, confusion, and classifier distortion.

Findings

Outperforms state-of-the-art exemplar-based methods by 18.24% on CIFAR-100.

Achieves 9.37% higher accuracy on ImageNet100.

Effectively mitigates catastrophic forgetting without using old class samples.

Abstract

Deep Neural Network (DNN) has achieved great success on datasets of closed class set. However, new classes, like new categories of social media topics, are continuously added to the real world, making it necessary to incrementally learn. This is hard for DNN because it tends to focus on fitting to new classes while ignoring old classes, a phenomenon known as catastrophic forgetting. State-of-the-art methods rely on knowledge distillation and data replay techniques but still have limitations. In this work, we analyze the causes of catastrophic forgetting in class incremental learning, which owes to three factors: representation drift, representation confusion, and classifier distortion. Based on this view, we propose a two-stage learning framework with a fixed encoder and an incrementally updated prototype classifier. The encoder is trained with self-supervised learning to generate a feature space with high intrinsic dimensionality, thus improving its transferability and generality. The classifier incrementally learns new prototypes while retaining the prototypes of previously learned data, which is crucial in preserving the decision boundary.Our method does not rely on preserved samples of old classes, is thus a non-exemplar based CIL method. Experiments on public datasets show that our method can significantly outperform state-of-the-art exemplar-based methods when they reserved 5 examplers per class, under the incremental setting of 10 phases, by 18.24% on CIFAR-100 and 9.37% on ImageNet100.