Estimating Uncertainty with Implicit Quantile Network

TL;DR

This paper introduces an Implicit Quantile Network approach for uncertainty quantification in predictions, providing a simple and effective alternative to traditional methods like ensembles and Bayesian neural networks, with promising results on image datasets.

Contribution

The paper proposes using Implicit Quantile Networks to directly model loss distribution for uncertainty estimation, offering a straightforward and effective method.

Findings

Incorrect predictions have twice the estimated loss

Removing high-uncertainty data improves accuracy by up to 10%

Method is simple to implement and useful for critical applications

Abstract

Uncertainty quantification is an important part of many performance critical applications. This paper provides a simple alternative to existing approaches such as ensemble learning and bayesian neural networks. By directly modeling the loss distribution with an Implicit Quantile Network, we get an estimate of how uncertain the model is of its predictions. For experiments with MNIST and CIFAR datasets, the mean of the estimated loss distribution is 2x higher for incorrect predictions. When data with high estimated uncertainty is removed from the test dataset, the accuracy of the model goes up as much as 10%. This method is simple to implement while offering important information to applications where the user has to know when the model could be wrong (e.g. deep learning for healthcare).