Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

TL;DR

This paper introduces a novel knowledge distillation method using conditional adversarial networks to efficiently train small, fast neural networks with improved accuracy for real-time applications.

Contribution

It proposes a new adversarial approach for knowledge transfer that enhances training of shallow, thin networks, especially effective for small student models.

Findings

Improved accuracy of small networks via adversarial knowledge distillation

Effective training of small networks with reduced inference time

Guidelines for selecting appropriate student network sizes

Abstract

There is an increasing interest on accelerating neural networks for real-time applications. We study the student-teacher strategy, in which a small and fast student network is trained with the auxiliary information learned from a large and accurate teacher network. We propose to use conditional adversarial networks to learn the loss function to transfer knowledge from teacher to student. The proposed method is particularly effective for relatively small student networks. Moreover, experimental results show the effect of network size when the modern networks are used as student. We empirically study the trade-off between inference time and classification accuracy, and provide suggestions on choosing a proper student network.