Class-Incremental Continual Learning into the eXtended DER-verse

TL;DR

This paper introduces eXtended-DER (X-DER), a continual learning method that improves knowledge retention and learning of new classes by revising replay memory and incorporating knowledge distillation, outperforming existing approaches.

Contribution

The paper proposes X-DER, an enhanced version of DER, with strategies for memory revision and learning unseen classes, achieving state-of-the-art results in class-incremental learning.

Findings

X-DER outperforms previous methods on CIFAR-100 and miniImageNet.

Memory revision and knowledge distillation significantly improve continual learning.

Extensive ablations confirm the effectiveness of proposed strategies.

Abstract

The staple of human intelligence is the capability of acquiring knowledge in a continuous fashion. In stark contrast, Deep Networks forget catastrophically and, for this reason, the sub-field of Class-Incremental Continual Learning fosters methods that learn a sequence of tasks incrementally, blending sequentially-gained knowledge into a comprehensive prediction. This work aims at assessing and overcoming the pitfalls of our previous proposal Dark Experience Replay (DER), a simple and effective approach that combines rehearsal and Knowledge Distillation. Inspired by the way our minds constantly rewrite past recollections and set expectations for the future, we endow our model with the abilities to i) revise its replay memory to welcome novel information regarding past data ii) pave the way for learning yet unseen classes. We show that the application of these strategies leads to remarkable improvements; indeed, the resulting method - termed eXtended-DER (X-DER) - outperforms the state of the art on both standard benchmarks (such as CIFAR-100 and miniImagenet) and a novel one here introduced. To gain a better understanding, we further provide extensive ablation studies that corroborate and extend the findings of our previous research (e.g. the value of Knowledge Distillation and flatter minima in continual learning setups).