Consistency Calibration: Improving Uncertainty Calibration via Consistency among Perturbed Neighbors

TL;DR

This paper introduces Consistency Calibration, a novel post-hoc method that improves uncertainty calibration in deep neural networks by leveraging model consistency across perturbed inputs, achieving state-of-the-art results without extra data or labels.

Contribution

The paper proposes a new calibration approach based on consistency among perturbed neighbors, offering an efficient, label-free, and effective alternative to traditional calibration methods.

Findings

Achieves state-of-the-art calibration performance on CIFAR-10, CIFAR-100, and ImageNet datasets.

Requires no additional data or labels, only input perturbations from source data.

Significantly improves calibration efficiency by perturbing at the logit level.

Abstract

Calibration is crucial in deep learning applications, especially in fields like healthcare and autonomous driving, where accurate confidence estimates are vital for decision-making. However, deep neural networks often suffer from miscalibration, with reliability diagrams and Expected Calibration Error (ECE) being the only standard perspective for evaluating calibration performance. In this paper, we introduce the concept of consistency as an alternative perspective on model calibration, inspired by uncertainty estimation literature in large language models (LLMs). We highlight its advantages over the traditional reliability-based view. Building on this concept, we propose a post-hoc calibration method called Consistency Calibration (CC), which adjusts confidence based on the model's consistency across perturbed inputs. CC is particularly effective in locally uncertainty estimation, as it requires no additional data samples or label information, instead generating input perturbations directly from the source data. Moreover, we show that performing perturbations at the logit level significantly improves computational efficiency. We validate the effectiveness of CC through extensive comparisons with various post-hoc and training-time calibration methods, demonstrating state-of-the-art performance on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet, as well as on long-tailed datasets like ImageNet-LT.