Condensed Prototype Replay for Class Incremental Learning

TL;DR

This paper introduces YONO, a novel class incremental learning method that replays only one condensed prototype per class, effectively reducing forgetting and outperforming traditional memory-based methods.

Contribution

YONO is the first approach to outperform memory-costly exemplar-replay methods using only one prototype per class, with a new prototype learning technique and an extension for synthetic data generation.

Findings

YONO outperforms existing IL methods in accuracy.

YONO significantly reduces catastrophic forgetting.

YONO+ further improves performance with synthetic data.

Abstract

Incremental learning (IL) suffers from catastrophic forgetting of old tasks when learning new tasks. This can be addressed by replaying previous tasks' data stored in a memory, which however is usually prone to size limits and privacy leakage. Recent studies store only class centroids as prototypes and augment them with Gaussian noises to create synthetic data for replay. However, they cannot effectively avoid class interference near their margins that leads to forgetting. Moreover, the injected noises distort the rich structure between real data and prototypes, hence even detrimental to IL. In this paper, we propose YONO that You Only Need to replay One condensed prototype per class, which for the first time can even outperform memory-costly exemplar-replay methods. To this end, we develop a novel prototype learning method that (1) searches for more representative prototypes in high-density regions by an attentional mean-shift algorithm and (2) moves samples in each class to their prototype to form a compact cluster distant from other classes. Thereby, the class margins are maximized, which effectively reduces interference causing future forgetting. In addition, we extend YONO to YONO+, which creates synthetic replay data by random sampling in the neighborhood of each prototype in the representation space. We show that the synthetic data can further improve YONO. Extensive experiments on IL benchmarks demonstrate the advantages of YONO/YONO+ over existing IL methods in terms of both accuracy and forgetting.