A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction

TL;DR

This paper introduces a Monte Carlo-based neural network framework for probabilistic sequence prediction from high-dimensional data, providing calibrated uncertainty estimates and confidence intervals, with applications demonstrated on synthetic and real datasets.

Contribution

The paper presents a novel Monte Carlo framework with a time-dependent regularization method for calibrated uncertainty estimation in sequence prediction.

Findings

Framework produces accurate discriminative predictions.

Regularization improves calibration of uncertainty estimates.

Effective on both synthetic and real data.

Abstract

Probabilistic prediction of sequences from images and other high-dimensional data is a key challenge, particularly in risk-sensitive applications. In these settings, it is often desirable to quantify the uncertainty associated with the prediction (instead of just determining the most likely sequence, as in language modeling). In this paper, we propose a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence. Our framework uses a Monte Carlo simulator, implemented as an autoregressively trained neural network, to sample sequences conditioned on an image input. We then use these samples to estimate the probabilities and confidence intervals. Experiments on synthetic and real data show that the framework produces accurate discriminative predictions, but can suffer from miscalibration. In order to address this shortcoming, we propose a time-dependent regularization method, which is shown to produce calibrated predictions.