Self-Distribution Distillation: Efficient Uncertainty Estimation

TL;DR

This paper introduces self-distribution distillation (S2D), a novel training method enabling a single model to efficiently estimate uncertainty, outperforming traditional methods and enabling resource-efficient ensemble approaches for safety-critical applications.

Contribution

The paper presents S2D, a new training approach that allows a single model to estimate uncertainty efficiently and can be combined into ensembles for improved performance.

Findings

S2D models outperform standard models and Monte-Carlo dropout on CIFAR-100.

S2D-based ensembles outperform standard deep ensembles in out-of-distribution detection.

Hierarchical ensemble distillation enhances uncertainty estimation efficiency.

Abstract

Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model's prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently train a single model that can estimate uncertainties. Furthermore it is possible to build ensembles of these models and apply hierarchical ensemble distillation approaches. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a standard deep ensemble can be outperformed using S2D based ensembles and novel distilled models.