Optimal Multiclass U-Calibration Error and Beyond

TL;DR

This paper determines the optimal bounds for online multiclass U-calibration error, showing it scales as the square root of the number of classes times the number of rounds, and explores improved bounds under specific loss conditions.

Contribution

It establishes the exact asymptotic optimal U-calibration error bounds for online multiclass prediction and verifies the optimality of a known algorithm through matching lower bounds.

Findings

Optimal U-calibration error is Θ(√KT).

Follow-the-Perturbed-Leader achieves this optimal bound.

Under certain conditions, U-calibration error can be reduced to logarithmic scale.

Abstract

We consider the problem of online multiclass U-calibration, where a forecaster aims to make sequential distributional predictions over $K$ classes with low U-calibration error, that is, low regret with respect to all bounded proper losses simultaneously. Kleinberg et al. (2023) developed an algorithm with U-calibration error $O(K\sqrt{T})$ after $T$ rounds and raised the open question of what the optimal bound is. We resolve this question by showing that the optimal U-calibration error is $\Theta(\sqrt{KT})$ -- we start with a simple observation that the Follow-the-Perturbed-Leader algorithm of Daskalakis and Syrgkanis (2016) achieves this upper bound, followed by a matching lower bound constructed with a specific proper loss (which, as a side result, also proves the optimality of the algorithm of Daskalakis and Syrgkanis (2016) in the context of online learning against an adversary with finite choices). We also strengthen our results under natural assumptions on the loss functions, including $\Theta(\log T)$ U-calibration error for Lipschitz proper losses, $O(\log T)$ U-calibration error for a certain class of decomposable proper losses, U-calibration error bounds for proper losses with a low covering number, and others.