Can You Trust This Prediction? Auditing Pointwise Reliability After Learning

TL;DR

This paper introduces Resampling Uncertainty Estimation (RUE), a post-training auditing method that assesses pointwise prediction reliability by estimating how predictions would vary with different training data, improving trust in high-stakes applications.

Contribution

RUE provides a novel post-training auditing algorithm that effectively detects unreliable predictions without requiring changes to the training process.

Findings

RUE outperforms existing reliability auditing tools in detecting inaccuracies.

RUE produces competitive predictive distributions compared to state-of-the-art methods.

RUE does not depend on specific training algorithms, making it broadly applicable.

Abstract

To use machine learning in high stakes applications (e.g. medicine), we need tools for building confidence in the system and evaluating whether it is reliable. Methods to improve model reliability often require new learning algorithms (e.g. using Bayesian inference to obtain uncertainty estimates). An alternative is to audit a model after it is trained. In this paper, we describe resampling uncertainty estimation (RUE), an algorithm to audit the pointwise reliability of predictions. Intuitively, RUE estimates the amount that a prediction would change if the model had been fit on different training data. The algorithm uses the gradient and Hessian of the model's loss function to create an ensemble of predictions. Experimentally, we show that RUE more effectively detects inaccurate predictions than existing tools for auditing reliability subsequent to training. We also show that RUE can create predictive distributions that are competitive with state-of-the-art methods like Monte Carlo dropout, probabilistic backpropagation, and deep ensembles, but does not depend on specific algorithms at train-time like these methods do.