Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model

TL;DR

This paper introduces DSKD, a novel knowledge distillation method that uses a diffusion model guided by the teacher classifier to improve feature alignment and transfer knowledge more effectively from teacher to student.

Contribution

The paper proposes a teacher-guided diffusion-based self-knowledge distillation approach that addresses feature distribution discrepancies, outperforming existing methods.

Findings

DSKD significantly outperforms existing KD methods across various models and datasets.

The diffusion-guided approach effectively reduces feature distribution mismatch.

Experimental results validate the superiority of DSKD in visual recognition tasks.

Abstract

Existing Knowledge Distillation (KD) methods often align feature information between teacher and student by exploring meaningful feature processing and loss functions. However, due to the difference in feature distributions between the teacher and student, the student model may learn incompatible information from the teacher. To address this problem, we propose teacher-guided student Diffusion Self-KD, dubbed as DSKD. Instead of the direct teacher-student alignment, we leverage the teacher classifier to guide the sampling process of denoising student features through a light-weight diffusion model. We then propose a novel locality-sensitive hashing (LSH)-guided feature distillation method between the original and denoised student features. The denoised student features encapsulate teacher knowledge and could be regarded as a teacher role. In this way, our DSKD method could eliminate discrepancies in mapping manners and feature distributions between the teacher and student, while learning meaningful knowledge from the teacher. Experiments on visual recognition tasks demonstrate that DSKD significantly outperforms existing KD methods across various models and datasets. Our code is attached in supplementary material.