MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator

TL;DR

MACEst is a novel model-agnostic method that provides reliable confidence estimates by explicitly accounting for both aleatoric and epistemic uncertainties, addressing flaws in standard approaches especially under high uncertainty.

Contribution

We introduce MACEst, a model-agnostic confidence estimator that estimates confidence locally and independently, improving trustworthiness over traditional global calibration methods.

Findings

MACEst yields more reliable confidence estimates in high uncertainty scenarios.

It outperforms existing calibration methods in trustworthiness.

The approach is applicable across different models and tasks.

Abstract

Reliable Confidence Estimates are hugely important for any machine learning model to be truly useful. In this paper, we argue that any confidence estimates based upon standard machine learning point prediction algorithms are fundamentally flawed and under situations with a large amount of epistemic uncertainty are likely to be untrustworthy. To address these issues, we present MACEst, a Model Agnostic Confidence Estimator, which provides reliable and trustworthy confidence estimates. The algorithm differs from current methods by estimating confidence independently as a local quantity which explicitly accounts for both aleatoric and epistemic uncertainty. This approach differs from standard calibration methods that use a global point prediction model as a starting point for the confidence estimate.