Discriminative Distillation to Reduce Class Confusion in Continual Learning

TL;DR

This paper introduces a discriminative distillation method to address class confusion in continual learning, improving recognition accuracy by focusing on discriminative features between similar classes, beyond just mitigating catastrophic forgetting.

Contribution

The study proposes a novel discriminative distillation strategy that enhances class distinction during continual learning, addressing class confusion issues not fully tackled by existing methods.

Findings

Improves continual learning performance when combined with existing methods.

Effectively reduces class confusion in natural image classification tasks.

Enhances discriminative feature learning between similar classes.

Abstract

Successful continual learning of new knowledge would enable intelligent systems to recognize more and more classes of objects. However, current intelligent systems often fail to correctly recognize previously learned classes of objects when updated to learn new classes. It is widely believed that such downgraded performance is solely due to the catastrophic forgetting of previously learned knowledge. In this study, we argue that the class confusion phenomena may also play a role in downgrading the classification performance during continual learning, i.e., the high similarity between new classes and any previously learned classes would also cause the classifier to make mistakes in recognizing these old classes, even if the knowledge of these old classes is not forgotten. To alleviate the class confusion issue, we propose a discriminative distillation strategy to help the classify well learn the discriminative features between confusing classes during continual learning. Experiments on multiple natural image classification tasks support that the proposed distillation strategy, when combined with existing methods, is effective in further improving continual learning.