Multi-Level Knowledge Distillation and Dynamic Self-Supervised Learning for Continual Learning

TL;DR

This paper introduces multi-level knowledge distillation and dynamic self-supervised learning techniques to enhance continual learning in class-incremental with repetition scenarios, leveraging unlabeled data for improved stability and plasticity.

Contribution

It proposes novel methods combining multi-level knowledge distillation and dynamic SSL to better utilize unlabeled data in continual learning, achieving state-of-the-art results.

Findings

Achieved 2nd place in the CVPR 5th CLVISION Challenge.

Significantly improved performance in class-incremental with repetition setup.

Demonstrated effective use of unlabeled data for continual learning.

Abstract

Class-incremental with repetition (CIR), where previously trained classes repeatedly introduced in future tasks, is a more realistic scenario than the traditional class incremental setup, which assumes that each task contains unseen classes. CIR assumes that we can easily access abundant unlabeled data from external sources, such as the Internet. Therefore, we propose two components that efficiently use the unlabeled data to ensure the high stability and the plasticity of models trained in CIR setup. First, we introduce multi-level knowledge distillation (MLKD) that distills knowledge from multiple previous models across multiple perspectives, including features and logits, so the model can maintain much various previous knowledge. Moreover, we implement dynamic self-supervised loss (SSL) to utilize the unlabeled data that accelerates the learning of new classes, while dynamic weighting of SSL keeps the focus of training to the primary task. Both of our proposed components significantly improve the performance in CIR setup, achieving 2nd place in the CVPR 5th CLVISION Challenge.