Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

TL;DR

This paper introduces Apprentice, a method that combines knowledge distillation with low-precision numerics to significantly enhance the accuracy of resource-efficient neural networks, achieving state-of-the-art results on ImageNet.

Contribution

It demonstrates how knowledge distillation can be effectively integrated with low-precision networks to improve their accuracy, providing three practical schemes for implementation.

Findings

Achieves state-of-the-art accuracy with ternary and 4-bit ResNet models.

Shows significant performance improvements over baseline low-precision networks.

Provides versatile schemes for applying knowledge distillation in training and deployment.

Abstract

Deep learning networks have achieved state-of-the-art accuracies on computer vision workloads like image classification and object detection. The performant systems, however, typically involve big models with numerous parameters. Once trained, a challenging aspect for such top performing models is deployment on resource constrained inference systems - the models (often deep networks or wide networks or both) are compute and memory intensive. Low-precision numerics and model compression using knowledge distillation are popular techniques to lower both the compute requirements and memory footprint of these deployed models. In this paper, we study the combination of these two techniques and show that the performance of low-precision networks can be significantly improved by using knowledge distillation techniques. Our approach, Apprentice, achieves state-of-the-art accuracies using ternary precision and 4-bit precision for variants of ResNet architecture on ImageNet dataset. We present three schemes using which one can apply knowledge distillation techniques to various stages of the train-and-deploy pipeline.