Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data

TL;DR

This paper introduces a feature affinity guided knowledge distillation method that enhances quantization-aware training of deep neural networks using unlabeled data, improving model compression efficiency.

Contribution

It proposes a novel FA-assisted KD approach with a fast approximation, enabling effective model compression on label-free data with reduced computational cost.

Findings

Enhanced model compression using unlabeled data.

FA loss improves intermediate feature learning.

FFA loss accelerates training for high-res images.

Abstract

In this paper, we propose a feature affinity (FA) assisted knowledge distillation (KD) method to improve quantization-aware training of deep neural networks (DNN). The FA loss on intermediate feature maps of DNNs plays the role of teaching middle steps of a solution to a student instead of only giving final answers in the conventional KD where the loss acts on the network logits at the output level. Combining logit loss and FA loss, we found that the quantized student network receives stronger supervision than from the labeled ground-truth data. The resulting FAQD is capable of compressing model on label-free data, which brings immediate practical benefits as pre-trained teacher models are readily available and unlabeled data are abundant. In contrast, data labeling is often laborious and expensive. Finally, we propose a fast feature affinity (FFA) loss that accurately approximates FA loss with a lower order of computational complexity, which helps speed up training for high resolution image input.