Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Loss

TL;DR

This paper explores the theoretical connections between reweighted risk functions like focal loss and calibration errors, proposing a flexible, efficient method that improves model calibration across various datasets.

Contribution

It establishes a principled link between calibration error and selective classification, introducing a flexible loss with a bin-based CDF approximation for better calibration.

Findings

Achieves competitive calibration performance on multiple datasets

Provides a flexible loss function with efficient gradient-based optimization

Demonstrates the theoretical connection between calibration error and selective classification

Abstract

Several variants of reweighted risk functionals, such as focal loss, inverse focal loss, and the Area Under the Risk Coverage Curve (AURC), have been proposed for improving model calibration; yet their theoretical connections to calibration errors remain under-explored. In this paper, we revisit a broad class of weighted risk functions and find a principled connection between calibration error and selective classification. We show that minimizing calibration error is closely linked to the selective classification paradigm and demonstrate that optimizing selective risk in low confidence regions naturally improves calibration. Our proposed loss shares a similar reweighting strategy with dual focal loss but offers greater flexibility through the choice of confidence score functions (CSFs). Furthermore, our approach utilizes a bin-based cumulative distribution function (CDF) approximation, enabling efficient gradient-based optimization with O(nM) complexity for n samples and M bins. Empirical evaluations demonstrate that our method achieves competitive calibration performance across a range of datasets and model architectures.