The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation

TL;DR

This paper explores the connection between multicalibration and property elicitation, establishing conditions under which multicalibrated predictors can be constructed and providing algorithms for various settings, with implications for fair risk assessment.

Contribution

It characterizes when multicalibration is possible via property elicitation and introduces algorithms for elicitable properties, extending previous work on means and quantiles.

Findings

Multicalibration is possible if and only if the property is elicitable.

For non-elicitable properties, even true predictors can fail to be calibrated.

Algorithms are provided for both batch and online settings for elicitable properties.

Abstract

We make a connection between multicalibration and property elicitation and show that (under mild technical conditions) it is possible to produce a multicalibrated predictor for a continuous scalar distributional property $\Gamma$ if and only if $\Gamma$ is elicitable. On the negative side, we show that for non-elicitable continuous properties there exist simple data distributions on which even the true distributional predictor is not calibrated. On the positive side, for elicitable $\Gamma$, we give simple canonical algorithms for the batch and the online adversarial setting, that learn a $\Gamma$-multicalibrated predictor. This generalizes past work on multicalibrated means and quantiles, and in fact strengthens existing online quantile multicalibration results. To further counter-weigh our negative result, we show that if a property $\Gamma^1$ is not elicitable by itself, but is elicitable conditionally on another elicitable property $\Gamma^0$, then there is a canonical algorithm that jointly multicalibrates $\Gamma^1$ and $\Gamma^0$; this generalizes past work on mean-moment multicalibration. Finally, as applications of our theory, we provide novel algorithmic and impossibility results for fair (multicalibrated) risk assessment.