Consistency Conditions for Differentiable Surrogate Losses

TL;DR

This paper investigates the relationship between calibration and indirect elicitation for differentiable surrogate losses, establishing conditions under which they are equivalent and introducing strong IE for better analysis.

Contribution

It extends the understanding of calibration equivalence from polyhedral to convex differentiable surrogates and introduces strong IE as a practical verification tool.

Findings

IE and calibration are equivalent for 1D convex differentiable losses.

Counter-example shows equivalence fails in higher dimensions.

Strong IE is necessary and sufficient for calibration in strongly convex cases.

Abstract

The statistical consistency of surrogate losses for discrete prediction tasks is often checked via the condition of calibration. However, directly verifying calibration can be arduous. Recent work shows that for polyhedral surrogates, a less arduous condition, indirect elicitation (IE), is still equivalent to calibration. We give the first results of this type for non-polyhedral surrogates, specifically the class of convex differentiable losses. We first prove that under mild conditions, IE and calibration are equivalent for one-dimensional losses in this class. We construct a counter-example that shows that this equivalence fails in higher dimensions. This motivates the introduction of strong IE, a strengthened form of IE that is equally easy to verify. We establish that strong IE implies calibration for differentiable surrogates and is both necessary and sufficient for strongly convex, differentiable surrogates. Finally, we apply these results to a range of problems to demonstrate the power of IE and strong IE for designing and analyzing consistent differentiable surrogates.