Quantifying Learning Guarantees for Convex but Inconsistent Surrogates

TL;DR

This paper extends the analysis of convex surrogate methods in machine learning to inconsistent cases, providing bounds on their effectiveness and implications for sample complexity and optimization, with applications to classification and ranking.

Contribution

It introduces a new lower bound on the calibration function for inconsistent surrogates, enabling quantification of their learning guarantees.

Findings

New lower bound on calibration function for inconsistent surrogates

Quantification of inconsistency effects on sample complexity

Application to multi-class classification and ranking tasks

Abstract

We study consistency properties of machine learning methods based on minimizing convex surrogates. We extend the recent framework of Osokin et al. (2017) for the quantitative analysis of consistency properties to the case of inconsistent surrogates. Our key technical contribution consists in a new lower bound on the calibration function for the quadratic surrogate, which is non-trivial (not always zero) for inconsistent cases. The new bound allows to quantify the level of inconsistency of the setting and shows how learning with inconsistent surrogates can have guarantees on sample complexity and optimization difficulty. We apply our theory to two concrete cases: multi-class classification with the tree-structured loss and ranking with the mean average precision loss. The results show the approximation-computation trade-offs caused by inconsistent surrogates and their potential benefits.