Knowledge Diffusion for Distillation

TL;DR

DiffKD introduces a diffusion-based method to explicitly denoise student features in knowledge distillation, effectively reducing noise and improving performance across multiple vision tasks.

Contribution

The paper proposes a novel diffusion model approach for knowledge distillation that explicitly denoises student features, outperforming existing methods.

Findings

Achieves state-of-the-art results on image classification, object detection, and segmentation.

Effective across various feature types and model capacities.

Reduces computational cost with a lightweight diffusion model.

Abstract

The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature, and propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher features. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https://github.com/hunto/DiffKD.