Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

TL;DR

This paper introduces a new metric called Conditional Congruence Error (CCE) to evaluate how well neural network regressors' predictive distributions match the true data distribution on a point-wise basis, addressing limitations of calibration metrics.

Contribution

The paper proposes the concept of conditional congruence and develops the CCE metric, providing a more precise assessment of probabilistic fit for neural regressors than existing calibration-based methods.

Findings

CCE demonstrates correctness, monotonicity, reliability, and robustness in high-dimensional regression tasks.

CCE outperforms traditional calibration metrics in diagnosing probabilistic misalignment.

The approach enables better understanding of individual input reliability in neural network predictions.

Abstract

While significant progress has been made in specifying neural networks capable of representing uncertainty, deep networks still often suffer from overconfidence and misaligned predictive distributions. Existing approaches for measuring this misalignment are primarily developed under the framework of calibration, with common metrics such as Expected Calibration Error (ECE). However, calibration can only provide a strictly marginal assessment of probabilistic alignment. Consequently, calibration metrics such as ECE are $\textit{distribution-wise}$ measures and cannot diagnose the $\textit{point-wise}$ reliability of individual inputs, which is important for real-world decision-making. We propose a stronger condition, which we term $\textit{conditional congruence}$, for assessing probabilistic fit. We also introduce a metric, Conditional Congruence Error (CCE), that uses conditional kernel mean embeddings to estimate the distance, at any point, between the learned predictive distribution and the empirical, conditional distribution in a dataset. We perform several high dimensional regression tasks and show that CCE exhibits four critical properties: $\textit{correctness}$, $\textit{monotonicity}$, $\textit{reliability}$, and $\textit{robustness}$.