Accuracy-Reliability Cost Function for Empirical Variance Estimation

TL;DR

This paper introduces an Accuracy-Reliability cost function for estimating variance in probabilistic forecasts, balancing accuracy and reliability, and demonstrates its effectiveness with neural networks and polynomial fits on synthetic data.

Contribution

The paper presents a novel cost function that jointly optimizes accuracy and reliability for variance estimation in heteroskedastic regression tasks.

Findings

Effectively recovers underlying noise functions in synthetic data

Applicable to one- and multi-dimensional problems

Implemented successfully with neural networks and polynomial fits

Abstract

In this paper we focus on the problem of assigning uncertainties to single-point predictions. We introduce a cost function that encodes the trade-off between accuracy and reliability in probabilistic forecast. We derive analytic formula for the case of forecasts of continuous scalar variables expressed in terms of Gaussian distributions. The Accuracy-Reliability cost function can be used to empirically estimate the variance in heteroskedastic regression problems (input dependent noise), by solving a two-objective optimization problem. The simple philosophy behind this strategy is that predictions based on the estimated variances should be both accurate and reliable (i.e. statistical consistent with observations). We show several examples with synthetic data, where the underlying hidden noise function can be accurately recovered, both in one and multi-dimensional problems. The practical implementation of the method has been done using a Neural Network and, in the one-dimensional case, with a simple polynomial fit.