UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations

TL;DR

UHKD introduces a novel frequency-domain approach for heterogeneous knowledge distillation, effectively transferring semantic information across different model architectures to improve compression without significant accuracy loss.

Contribution

The paper proposes a unified framework leveraging frequency-domain features and alignment modules to enhance heterogeneous knowledge distillation, addressing limitations of prior methods focused on logits.

Findings

Achieves up to 5.59% accuracy improvement on CIFAR-100.

Achieves up to 0.83% accuracy improvement on ImageNet-1K.

Effective in transferring semantic knowledge across diverse architectures.

Abstract

Knowledge distillation (KD) is an effective model compression technique that transfers knowledge from a high-performance teacher to a lightweight student, reducing computational and storage costs while maintaining competitive accuracy. However, most existing KD methods are tailored for homogeneous models and perform poorly in heterogeneous settings, particularly when intermediate features are involved. Semantic discrepancies across architectures hinder effective use of intermediate representations from the teacher model, while prior heterogeneous KD studies mainly focus on the logits space, underutilizing rich semantic information in intermediate layers. To address this, Unified Heterogeneous Knowledge Distillation (UHKD) is proposed, a framework that leverages intermediate features in the frequency domain for cross-architecture transfer. Frequency-domain representations are leveraged to capture global semantic knowledge and mitigate representational discrepancies between heterogeneous teacher-student pairs. Specifically, a Feature Transformation Module (FTM) generates compact frequency-domain representations of teacher features, while a learnable Feature Alignment Module (FAM) projects student features and aligns them via multi-level matching. Training is guided by a joint objective combining mean squared error on intermediate features with Kullback-Leibler divergence on logits. Extensive experiments on CIFAR-100 and ImageNet-1K demonstrate the effectiveness of the proposed approach, achieving maximum gains of 5.59% and 0.83% over the latest heterogeneous distillation method on the two datasets, respectively. Code will be released soon.