Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond

TL;DR

This paper introduces DiffFR, a diffusion-based feature replay method for non-exemplar class-incremental learning, which generates highly similar class features to mitigate catastrophic forgetting without storing old samples.

Contribution

It proposes a novel diffusion model approach combined with Siamese self-supervised learning and prototype calibration for improved NECIL performance.

Findings

DiffFR outperforms state-of-the-art NECIL methods by 3.0% on average.

Diffusion models generate class-representative features closely matching real features.

The approach effectively mitigates catastrophic forgetting without storing old class samples.

Abstract

Non-exemplar class-incremental learning (NECIL) is to resist catastrophic forgetting without saving old class samples. Prior methodologies generally employ simple rules to generate features for replaying, suffering from large distribution gap between replayed features and real ones. To address the aforementioned issue, we propose a simple, yet effective \textbf{Diff}usion-based \textbf{F}eature \textbf{R}eplay (\textbf{DiffFR}) method for NECIL. First, to alleviate the limited representational capacity caused by fixing the feature extractor, we employ Siamese-based self-supervised learning for initial generalizable features. Second, we devise diffusion models to generate class-representative features highly similar to real features, which provides an effective way for exemplar-free knowledge memorization. Third, we introduce prototype calibration to direct the diffusion model's focus towards learning the distribution shapes of features, rather than the entire distribution. Extensive experiments on public datasets demonstrate significant performance gains of our DiffFR, outperforming the state-of-the-art NECIL methods by 3.0\% in average. The code will be made publicly available soon.