On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning

TL;DR

This paper introduces a novel knowledge distillation method called Displacement Knowledge Distillation (DKD) that enhances few-shot class-incremental learning by incorporating displacement information, and proposes a dual network architecture to improve performance on evolving data distributions.

Contribution

The paper presents DKD, a new distillation technique using displacement instead of similarity, and DDNet, a dual network architecture for better handling base and novel classes in FSCIL.

Findings

DDNet achieves state-of-the-art results on three benchmarks.

DKD improves knowledge retention with displacement-based distillation.

The approach demonstrates robustness and general applicability.

Abstract

Few-shot Class-Incremental Learning (FSCIL) addresses the challenges of evolving data distributions and the difficulty of data acquisition in real-world scenarios. To counteract the catastrophic forgetting typically encountered in FSCIL, knowledge distillation is employed as a way to maintain the knowledge from learned data distribution. Recognizing the limitations of generating discriminative feature representations in a few-shot context, our approach incorporates structural information between samples into knowledge distillation. This structural information serves as a remedy for the low quality of features. Diverging from traditional structured distillation methods that compute sample similarity, we introduce the Displacement Knowledge Distillation (DKD) method. DKD utilizes displacement rather than similarity between samples, incorporating both distance and angular information to significantly enhance the information density retained through knowledge distillation. Observing performance disparities in feature distribution between base and novel classes, we propose the Dual Distillation Network (DDNet). This network applies traditional knowledge distillation to base classes and DKD to novel classes, challenging the conventional integration of novel classes with base classes. Additionally, we implement an instance-aware sample selector during inference to dynamically adjust dual branch weights, thereby leveraging the complementary strengths of each approach. Extensive testing on three benchmarks demonstrates that DDNet achieves state-of-the-art results. Moreover, through rigorous experimentation and comparison, we establish the robustness and general applicability of our proposed DKD method.