Surrogate Regret Bounds for Polyhedral Losses

TL;DR

This paper establishes general regret bounds for surrogate losses in machine learning, showing polyhedral surrogates offer optimal linear rates, while non-polyhedral ones have slower square root rates, impacting surrogate choice.

Contribution

It provides the first general linear regret bounds for polyhedral surrogates and characterizes slower rates for non-polyhedral surrogates, advancing understanding of surrogate loss effectiveness.

Findings

Polyhedral surrogates admit linear regret bounds.

Non-polyhedral surrogates have square root regret bounds.

Polyhedral surrogates are often optimal for generalization.

Abstract

Surrogate risk minimization is an ubiquitous paradigm in supervised machine learning, wherein a target problem is solved by minimizing a surrogate loss on a dataset. Surrogate regret bounds, also called excess risk bounds, are a common tool to prove generalization rates for surrogate risk minimization. While surrogate regret bounds have been developed for certain classes of loss functions, such as proper losses, general results are relatively sparse. We provide two general results. The first gives a linear surrogate regret bound for any polyhedral (piecewise-linear and convex) surrogate, meaning that surrogate generalization rates translate directly to target rates. The second shows that for sufficiently non-polyhedral surrogates, the regret bound is a square root, meaning fast surrogate generalization rates translate to slow rates for the target. Together, these results suggest polyhedral surrogates are optimal in many cases.