Evaluating and Calibrating Uncertainty Prediction in Regression Tasks

TL;DR

This paper examines the calibration of uncertainty predictions in regression tasks, identifies limitations in existing definitions, and proposes a new evaluation and calibration method validated on synthetic and real-world datasets.

Contribution

It introduces a new definition for calibration of regression uncertainty and a simple histogram-based evaluation method, along with an effective scaling calibration approach.

Findings

Existing calibration definitions are limited in distinguishing informative uncertainty.

The proposed histogram-based evaluation effectively assesses calibration.

Scaling calibration performs comparably to complex methods on real datasets.

Abstract

Predicting not only the target but also an accurate measure of uncertainty is important for many machine learning applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for calibration of a regression uncertainty [Kuleshov et al. 2018] has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple, scaling-based calibration method that preforms as well as much more complex ones. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO and KITTI datasets.