Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation

TL;DR

This paper introduces a novel knowledge distillation approach that uses an assistant model to bridge heterogeneous teacher-student pairs, employing a spatial-agnostic loss to improve feature alignment and achieve state-of-the-art results.

Contribution

It proposes a new framework for cross-architecture knowledge distillation using an assistant model and a spatial-agnostic loss, enabling effective transfer between diverse model types.

Findings

Achieves up to 11.47% accuracy gain on CIFAR-100.

Improves performance on ImageNet-1K by 3.67%.

Effective across CNNs, ViTs, and MLPs.

Abstract

Most knowledge distillation (KD) methodologies predominantly focus on teacher-student pairs with similar architectures, such as both being convolutional neural networks (CNNs). However, the potential and flexibility of KD can be greatly improved by expanding it to novel Cross-Architecture KD (CAKD), where the knowledge of homogeneous and heterogeneous teachers can be transferred flexibly to a given student. The primary challenge in CAKD lies in the substantial feature gaps between heterogeneous models, originating from the distinction of their inherent inductive biases and module functions. To this end, we introduce an assistant model as a bridge to facilitate smooth feature knowledge transfer between heterogeneous teachers and students. More importantly, within our proposed design principle, the assistant model combines the advantages of cross-architecture inductive biases and module functions by merging convolution and attention modules derived from both student and teacher module functions. Furthermore, we observe that heterogeneous features exhibit diverse spatial distributions in CAKD, hindering the effectiveness of conventional pixel-wise mean squared error (MSE) loss. Therefore, we leverage a spatial-agnostic InfoNCE loss to align features after spatial smoothing, thereby improving the feature alignments in CAKD. Our proposed method is evaluated across some homogeneous model pairs and arbitrary heterogeneous combinations of CNNs, ViTs, and MLPs, achieving state-of-the-art performance for distilled models with a maximum gain of 11.47% on CIFAR-100 and 3.67% on ImageNet-1K. Our code and models will be released.