Adaptive Calibration in Non-Stationary Environments

TL;DR

This paper develops adaptive online prediction algorithms that automatically adjust calibration error bounds based on the environment's non-stationarity, smoothly transitioning between stationary and adversarial settings.

Contribution

The authors introduce algorithms with calibration guarantees that adapt to the environment's non-stationarity, extending prior work with epoch-based scheduling and a novel partitioning method.

Findings

Achieve calibration error bounds of tilde;(\u221a{T}+(TC)^{1/3}) for alibration.

Attain bounds of tilde;((1+C)^{1/3}) for  and pseudo KL calibration.

Bounds match optimal rates in stationary and adversarial regimes.

Abstract

Making calibrated online predictions is a central challenge in modern AI systems. Much of the existing literature focuses on fully adversarial environments where outcomes may be arbitrary, leading to conservative algorithms that can perform suboptimally in more benign settings, such as when outcomes are nearly stationary. This gap raises a natural question: can we design online prediction algorithms whose calibration error automatically adapts to the degree of non-stationarity in the environment, smoothly interpolating between i.i.d. and adversarial regimes? We answer this question in the affirmative and develop a suite of algorithms that achieve adaptive calibration guarantees under multiple calibration measures. Specifically, with $T$ being the number of rounds and $C\in[0,T]$ being an unknown non-stationary measure defined as the minimal $\ell_1$ deviation of the mean outcomes, our algorithms attain $\widetilde{O}(\sqrt{T}+(TC)^{\frac{1}{3}})$ for $\ell_1$ calibration error and $\widetilde{O}((1+C)^{\frac{1}{3}})$ for both $\ell_2$ and pseudo KL calibration error. These bounds match the optimal rates in the stationary case ($C=0$) and recover known guarantees in the fully adversarial regime ($C=T$). Our approach builds on and extends prior work [Hu et al., 2026, Luo et al., 2025], introducing an epoch-based scheduling together with a novel non-uniform partition of the prediction space that allocates finer resolution near the underlying ground truth.