Class incremental learning with probability dampening and cascaded gated classifier

TL;DR

This paper introduces a novel regularization method and a cascaded gated classifier to improve continual learning in neural networks, reducing forgetting without extensive replay buffers.

Contribution

It proposes Margin Dampening and Cascaded Scaling Classifier, novel techniques that enhance knowledge retention and task adaptation in class incremental learning.

Findings

Outperforms baseline methods on multiple benchmarks.

Effectively mitigates forgetting without large memory buffers.

Analyzes component contributions to overall performance.

Abstract

Humans are capable of acquiring new knowledge and transferring learned knowledge into different domains, incurring a small forgetting. The same ability, called Continual Learning, is challenging to achieve when operating with neural networks due to the forgetting affecting past learned tasks when learning new ones. This forgetting can be mitigated by replaying stored samples from past tasks, but a large memory size may be needed for long sequences of tasks; moreover, this could lead to overfitting on saved samples. In this paper, we propose a novel regularisation approach and a novel incremental classifier called, respectively, Margin Dampening and Cascaded Scaling Classifier. The first combines a soft constraint and a knowledge distillation approach to preserve past learned knowledge while allowing the model to learn new patterns effectively. The latter is a gated incremental classifier, helping the model modify past predictions without directly interfering with them. This is achieved by modifying the output of the model with auxiliary scaling functions. We empirically show that our approach performs well on multiple benchmarks against well-established baselines, and we also study each component of our proposal and how the combinations of such components affect the final results.