h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective

TL;DR

This paper introduces h-calibration, a probabilistic framework for classifier recalibration that addresses limitations of existing methods, achieving state-of-the-art results in producing reliable probability estimates for neural networks.

Contribution

The paper proposes a novel probabilistic learning framework for calibration, overcoming ten common limitations of previous methods and providing a simple, effective post-hoc calibration algorithm with theoretical guarantees.

Findings

Achieves state-of-the-art calibration performance on benchmarks.

Theoretically constructs an error-bounded calibration objective.

Demonstrates improved reliability of neural network probabilities.

Abstract

Deep neural networks have demonstrated remarkable performance across numerous learning tasks but often suffer from miscalibration, resulting in unreliable probability outputs. This has inspired many recent works on mitigating miscalibration, particularly through post-hoc recalibration methods that aim to obtain calibrated probabilities without sacrificing the classification performance of pre-trained models. In this study, we summarize and categorize previous works into three general strategies: intuitively designed methods, binning-based methods, and methods based on formulations of ideal calibration. Through theoretical and practical analysis, we highlight ten common limitations in previous approaches. To address these limitations, we propose a probabilistic learning framework for calibration called h-calibration, which theoretically constructs an equivalent learning formulation for canonical calibration with boundedness. On this basis, we design a simple yet effective post-hoc calibration algorithm. Our method not only overcomes the ten identified limitations but also achieves markedly better performance than traditional methods, as validated by extensive experiments. We further analyze, both theoretically and experimentally, the relationship and advantages of our learning objective compared to traditional proper scoring rule. In summary, our probabilistic framework derives an approximately equivalent differentiable objective for learning error-bounded calibrated probabilities, elucidating the correspondence and convergence properties of computational statistics with respect to theoretical bounds in canonical calibration. The theoretical effectiveness is verified on standard post-hoc calibration benchmarks by achieving state-of-the-art performance. This research offers valuable reference for learning reliable likelihood in related fields.