Neural Collapse Inspired Knowledge Distillation

TL;DR

This paper introduces a novel knowledge distillation method inspired by Neural Collapse, which enhances student model performance by transferring the teacher's geometric feature structure, leading to improved generalization and state-of-the-art results.

Contribution

The paper proposes a new distillation paradigm that incorporates Neural Collapse structure transfer, offering a more effective way to bridge the knowledge gap between teacher and student.

Findings

NCKD improves student model accuracy.

Transferring NC structure enhances generalization.

Achieves state-of-the-art performance.

Abstract

Existing knowledge distillation (KD) methods have demonstrated their ability in achieving student network performance on par with their teachers. However, the knowledge gap between the teacher and student remains significant and may hinder the effectiveness of the distillation process. In this work, we introduce the structure of Neural Collapse (NC) into the KD framework. NC typically occurs in the final phase of training, resulting in a graceful geometric structure where the last-layer features form a simplex equiangular tight frame. Such phenomenon has improved the generalization of deep network training. We hypothesize that NC can also alleviate the knowledge gap in distillation, thereby enhancing student performance. This paper begins with an empirical analysis to bridge the connection between knowledge distillation and neural collapse. Through this analysis, we establish that transferring the teacher's NC structure to the student benefits the distillation process. Therefore, instead of merely transferring instance-level logits or features, as done by existing distillation methods, we encourage students to learn the teacher's NC structure. Thereby, we propose a new distillation paradigm termed Neural Collapse-inspired Knowledge Distillation (NCKD). Comprehensive experiments demonstrate that NCKD is simple yet effective, improving the generalization of all distilled student models and achieving state-of-the-art accuracy performance.