Novel Uncertainty Framework for Deep Learning Ensembles

TL;DR

This paper introduces a new uncertainty estimation framework for deep learning ensembles based on statistical mechanics, enhancing reliability and achieving state-of-the-art results in classification tasks.

Contribution

It proposes a novel statistical mechanics framework for dropout, providing improved uncertainty estimates applicable across various neural network architectures.

Findings

Achieved state-of-the-art AUC on benchmark datasets.

Enabled 'don't-know' responses to improve classifier reliability.

Demonstrated broad applicability to different deep learning tasks.

Abstract

Deep neural networks have become the default choice for many of the machine learning tasks such as classification and regression. Dropout, a method commonly used to improve the convergence of deep neural networks, generates an ensemble of thinned networks with extensive weight sharing. Recent studies that dropout can be viewed as an approximate variational inference in Gaussian processes, and used as a practical tool to obtain uncertainty estimates of the network. We propose a novel statistical mechanics based framework to dropout and use this framework to propose a new generic algorithm that focuses on estimates of the variance of the loss as measured by the ensemble of thinned networks. Our approach can be applied to a wide range of deep neural network architectures and machine learning tasks. In classification, this algorithm allows the generation of a don't-know answer to be generated, which can increase the reliability of the classifier. Empirically we demonstrate state-of-the-art AUC results on publicly available benchmarks.