A Universal Growth Rate for Learning with Smooth Surrogate Losses

TL;DR

This paper establishes a universal square-root growth rate for smooth surrogate losses in classification, providing new bounds and insights for both binary and multi-class scenarios, aiding in loss selection.

Contribution

It proves a universal square-root growth rate for smooth surrogate losses in classification and analyzes minimizability gaps to guide loss selection.

Findings

Square-root growth rate near zero for smooth margin-based surrogate losses

New upper and lower bounds for excess error bounds

Analysis of minimizability gaps for loss comparison

Abstract

This paper presents a comprehensive analysis of the growth rate of $H$-consistency bounds (and excess error bounds) for various surrogate losses used in classification. We prove a square-root growth rate near zero for smooth margin-based surrogate losses in binary classification, providing both upper and lower bounds under mild assumptions. This result also translates to excess error bounds. Our lower bound requires weaker conditions than those in previous work for excess error bounds, and our upper bound is entirely novel. Moreover, we extend this analysis to multi-class classification with a series of novel results, demonstrating a universal square-root growth rate for smooth comp-sum and constrained losses, covering common choices for training neural networks in multi-class classification. Given this universal rate, we turn to the question of choosing among different surrogate losses. We first examine how $H$-consistency bounds vary across surrogates based on the number of classes. Next, ignoring constants and focusing on behavior near zero, we identify minimizability gaps as the key differentiating factor in these bounds. Thus, we thoroughly analyze these gaps, to guide surrogate loss selection, covering: comparisons across different comp-sum losses, conditions where gaps become zero, and general conditions leading to small gaps. Additionally, we demonstrate the key role of minimizability gaps in comparing excess error bounds and $H$-consistency bounds.