The Minimax Rate of Second-Order Calibration

TL;DR

This paper establishes the optimal rate for estimating second-order calibration error in binary classification, improving previous methods and providing finite-sample guarantees for recalibration procedures.

Contribution

It characterizes the minimax rate of second-order calibration error estimation and introduces a bucket-free definition, with practical recalibration guarantees.

Findings

Polynomial regression achieves rate  O(1/ ) with explicit constants.

A matching lower bound confirms the minimax optimality of the rate.

Experiments validate the theoretical rate and improved uncertainty calibration.

Abstract

We characterize the minimax rate of estimating the second-order calibration error for binary classification, which quantifies whether a higher-order predictor's epistemic-uncertainty estimate matches the conditional variance of the label probability on its level sets. Our key observation is that the sech perturbation kernel, previously used only to enforce smoothness of calibration functions, in fact makes them analytic in a strip of half-width $h\pi/2$. Polynomial regression then estimates the calibration error at rate $\tilde{O}(1/\sqrt{n})$, with explicit constants, a qualitative improvement over the $O(n^{-1/4})$ rate achievable by bucketing or kernel smoothing. A matching $\Omega(1/\sqrt{n})$ lower bound establishes minimax optimality up to logarithmic factors. As a corollary, we give the first finite-sample guarantee for second-order Platt scaling, yielding a post-hoc procedure that recalibrates both the mean prediction and the epistemic-variance estimate of any higher-order predictor. Along the way, we provide a bucket-free definition of second-order calibration and relate it quantitatively to the bucketed formulation of Ahdritz et al. [2025]. Our experiments confirm the predicted rate and the quality of the recalibrated uncertainties.