Class-incremental Learning using a Sequence of Partial Implicitly Regularized Classifiers

TL;DR

This paper introduces a novel class-incremental learning method that trains multiple specialized classifiers to reduce catastrophic forgetting, demonstrating significant improvements over state-of-the-art techniques on CIFAR100.

Contribution

It proposes training multiple non-interfering classifiers instead of a single one, enhancing incremental learning performance with a sequence of partial implicit regularization.

Findings

Significant performance improvement over SOTA on CIFAR100

Specialized classifiers reduce interference and forgetting

Method effectively leverages pretrained feature extractors

Abstract

In class-incremental learning, the objective is to learn a number of classes sequentially without having access to the whole training data. However, due to a problem known as catastrophic forgetting, neural networks suffer substantial performance drop in such settings. The problem is often approached by experience replay, a method which stores a limited number of samples to be replayed in future steps to reduce forgetting of the learned classes. When using a pretrained network as a feature extractor, we show that instead of training a single classifier incrementally, it is better to train a number of specialized classifiers which do not interfere with each other yet can cooperatively predict a single class. Our experiments on CIFAR100 dataset show that the proposed method improves the performance over SOTA by a large margin.