Simple Regularisation for Uncertainty-Aware Knowledge Distillation

TL;DR

This paper introduces a simple regularisation method for distilling ensemble neural networks into a single model, preserving their uncertainty estimation and accuracy without complex fine-tuning, applicable across various datasets and architectures.

Contribution

A novel regularisation technique for distribution-free knowledge distillation that maintains ensemble-like uncertainty and accuracy in a single neural network.

Findings

Effective across toy data, SVHN, CIFAR-10

Preserves diversity and uncertainty estimation

Applicable to various neural network architectures

Abstract

Considering uncertainty estimation of modern neural networks (NNs) is one of the most important steps towards deploying machine learning systems to meaningful real-world applications such as in medicine, finance or autonomous systems. At the moment, ensembles of different NNs constitute the state-of-the-art in both accuracy and uncertainty estimation in different tasks. However, ensembles of NNs are unpractical under real-world constraints, since their computation and memory consumption scale linearly with the size of the ensemble, which increase their latency and deployment cost. In this work, we examine a simple regularisation approach for distribution-free knowledge distillation of ensemble of machine learning models into a single NN. The aim of the regularisation is to preserve the diversity, accuracy and uncertainty estimation characteristics of the original ensemble without any intricacies, such as fine-tuning. We demonstrate the generality of the approach on combinations of toy data, SVHN/CIFAR-10, simple to complex NN architectures and different tasks.