Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures

TL;DR

This paper introduces MLDR-KD, a novel relational distillation framework that enhances knowledge transfer across heterogeneous neural network architectures by balancing dark knowledge and confidence, leading to significant performance improvements.

Contribution

The paper proposes a multi-level decoupled relational distillation method with dynamic feature fusion, advancing heterogeneous model distillation beyond existing approaches.

Findings

Achieves up to 4.86% accuracy gain on CIFAR-100

Improves performance on Tiny-ImageNet by 2.78%

Demonstrates robustness across diverse architectures

Abstract

Heterogeneous distillation is an effective way to transfer knowledge from cross-architecture teacher models to student models. However, existing heterogeneous distillation methods do not take full advantage of the dark knowledge hidden in the teacher's output, limiting their performance.To this end, we propose a novel framework named Multi-Level Decoupled Relational Knowledge Distillation (MLDR-KD) to unleash the potential of relational distillation in heterogeneous distillation. Concretely, we first introduce Decoupled Finegrained Relation Alignment (DFRA) in both logit and feature levels to balance the trade-off between distilled dark knowledge and the confidence in the correct category of the heterogeneous teacher model. Then, Multi-Scale Dynamic Fusion (MSDF) module is applied to dynamically fuse the projected logits of multiscale features at different stages in student model, further improving performance of our method in feature level. We verify our method on four architectures (CNNs, Transformers, MLPs and Mambas), two datasets (CIFAR-100 and Tiny-ImageNet). Compared with the best available method, our MLDR-KD improves student model performance with gains of up to 4.86% on CIFAR-100 and 2.78% on Tiny-ImageNet datasets respectively, showing robustness and generality in heterogeneous distillation. Code will be released soon.