Fundamental Novel Consistency Theory: $H$-Consistency Bounds

TL;DR

This paper introduces $H$-consistency bounds that provide stronger, more informative guarantees on the estimation error between surrogate and target losses in machine learning, applicable to various classification settings and loss functions.

Contribution

It develops a comprehensive framework for deriving $H$-consistency bounds for multiple loss functions, including new bounds for multi-class and comp-sum losses, advancing theoretical understanding of surrogate-target loss relationships.

Findings

Established tight distribution-dependent and -independent bounds for binary classification.

Derived the first $H$-consistency bounds for multi-class max, sum, and constrained losses.

Proposed smooth adversarial variants of losses for robust learning.

Abstract

In machine learning, the loss functions optimized during training often differ from the target loss that defines task performance due to computational intractability or lack of differentiability. We present an in-depth study of the target loss estimation error relative to the surrogate loss estimation error. Our analysis leads to $H$-consistency bounds, which are guarantees accounting for the hypothesis set $H$. These bounds offer stronger guarantees than Bayes-consistency or $H$-calibration and are more informative than excess error bounds. We begin with binary classification, establishing tight distribution-dependent and -independent bounds. We provide explicit bounds for convex surrogates (including linear models and neural networks) and analyze the adversarial setting for surrogates like $\rho$-margin and sigmoid loss. Extending to multi-class classification, we present the first $H$-consistency bounds for max, sum, and constrained losses, covering both non-adversarial and adversarial scenarios. We demonstrate that in some cases, non-trivial $H$-consistency bounds are unattainable. We also investigate comp-sum losses (e.g., cross-entropy, MAE), deriving their first $H$-consistency bounds and introducing smooth adversarial variants that yield robust learning algorithms. We develop a comprehensive framework for deriving these bounds across various surrogates, introducing new characterizations for constrained and comp-sum losses. Finally, we examine the growth rates of $H$-consistency bounds, establishing a universal square-root growth rate for smooth surrogates in binary and multi-class tasks, and analyze minimizability gaps to guide surrogate selection.