Simple Calibration via Geodesic Kernels

TL;DR

This paper introduces a novel calibration method using geodesic kernels that improves the reliability of deep models' confidence estimates across both in-distribution and out-of-distribution data, crucial for safety-critical applications.

Contribution

It proposes a new calibration approach leveraging geodesic distances and Gaussian kernels within the polytopic partitions learned by deep models, addressing both ID and OOD calibration challenges.

Findings

KDF and KDN achieve well-calibrated posteriors for ID and OOD data.

The methods preserve classification accuracy while extrapolating beyond training data.

Experiments on tabular and vision benchmarks validate effectiveness.

Abstract

Deep discriminative approaches, such as decision forests and deep neural networks, have recently found applications in many important real-world scenarios. However, deploying these learning algorithms in safety-critical applications raises concerns, particularly when it comes to ensuring calibration for both in-distribution and out-of-distribution regions. Many popular methods for in-distribution (ID) calibration, such as isotonic and Platt's sigmoidal regression, exhibit adequate ID calibration performance. However, these methods are not calibrated for the entire feature space, leading to overconfidence in the out-of-distribution (OOD) region. Existing OOD calibration methods generally exhibit poor ID calibration. In this paper, we jointly address the ID and OOD problems. We leveraged the fact that deep models learn to partition feature space into a union of polytopes, that is, flat-sided geometric objects. We introduce a geodesic distance to measure the distance between these polytopes and further distinguish samples within the same polytope using a Gaussian kernel. Our experiments on both tabular and vision benchmarks show that the proposed approaches, namely Kernel Density Forest (KDF) and Kernel Density Network (KDN), obtain well-calibrated posteriors for both ID and OOD samples, while mostly preserving the classification accuracy and extrapolating beyond the training data to handle OOD inputs appropriately.