Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks

TL;DR

This paper introduces a novel framework combining multi-teacher knowledge distillation and network quantization to effectively train low bit-width deep neural networks, enabling collaborative learning among quantized models.

Contribution

It proposes a new method that integrates multi-teacher knowledge distillation with network quantization, fostering collaborative and mutual learning among quantized teachers and students.

Findings

Quantized student models outperform some full-precision models.

The method achieves competitive results on CIFAR100 and ImageNet.

Collaborative learning enhances low bit-width DNN performance.

Abstract

Knowledge distillation which learns a lightweight student model by distilling knowledge from a cumbersome teacher model is an attractive approach for learning compact deep neural networks (DNNs). Recent works further improve student network performance by leveraging multiple teacher networks. However, most of the existing knowledge distillation-based multi-teacher methods use separately pretrained teachers. This limits the collaborative learning between teachers and the mutual learning between teachers and student. Network quantization is another attractive approach for learning compact DNNs. However, most existing network quantization methods are developed and evaluated without considering multi-teacher support to enhance the performance of quantized student model. In this paper, we propose a novel framework that leverages both multi-teacher knowledge distillation and network quantization for learning low bit-width DNNs. The proposed method encourages both collaborative learning between quantized teachers and mutual learning between quantized teachers and quantized student. During learning process, at corresponding layers, knowledge from teachers will form an importance-aware shared knowledge which will be used as input for teachers at subsequent layers and also be used to guide student. Our experimental results on CIFAR100 and ImageNet datasets show that the compact quantized student models trained with our method achieve competitive results compared to other state-of-the-art methods, and in some cases, indeed surpass the full precision models.