Improving the Reliability for Confidence Estimation

TL;DR

This paper introduces a meta-learning framework that enhances confidence estimation models to better handle label imbalance and out-of-distribution data, improving their reliability in diverse deployment scenarios.

Contribution

It proposes a novel meta-learning approach with virtual training and testing sets to improve confidence estimation under distribution shifts.

Findings

Improved confidence estimation in monocular depth estimation.

Enhanced robustness to out-of-distribution inputs in image classification.

Framework generalizes well across different tasks.

Abstract

Confidence estimation, a task that aims to evaluate the trustworthiness of the model's prediction output during deployment, has received lots of research attention recently, due to its importance for the safe deployment of deep models. Previous works have outlined two important qualities that a reliable confidence estimation model should possess, i.e., the ability to perform well under label imbalance and the ability to handle various out-of-distribution data inputs. In this work, we propose a meta-learning framework that can simultaneously improve upon both qualities in a confidence estimation model. Specifically, we first construct virtual training and testing sets with some intentionally designed distribution differences between them. Our framework then uses the constructed sets to train the confidence estimation model through a virtual training and testing scheme leading it to learn knowledge that generalizes to diverse distributions. We show the effectiveness of our framework on both monocular depth estimation and image classification.