On the Error Resistance of Hinge Loss Minimization

TL;DR

This paper proves that hinge loss minimization algorithms like SVMs are robust to label errors under certain data conditions, such as linear separability with a margin and near isotropic logconcave distributions.

Contribution

It establishes theoretical robustness guarantees for hinge loss minimization under specific data assumptions and adversarial label noise.

Findings

Robustness of hinge loss minimization with linear margin and near isotropic logconcave data.

Negligible error on uncorrupted data despite constant fraction of adversarial label noise.

Unified framework for understanding surrogate loss robustness in classification.

Abstract

Commonly used classification algorithms in machine learning, such as support vector machines, minimize a convex surrogate loss on training examples. In practice, these algorithms are surprisingly robust to errors in the training data. In this work, we identify a set of conditions on the data under which such surrogate loss minimization algorithms provably learn the correct classifier. This allows us to establish, in a unified framework, the robustness of these algorithms under various models on data as well as error. In particular, we show that if the data is linearly classifiable with a slightly non-trivial margin (i.e. a margin at least $C/\sqrt{d}$ for $d$-dimensional unit vectors), and the class-conditional distributions are near isotropic and logconcave, then surrogate loss minimization has negligible error on the uncorrupted data even when a constant fraction of examples are adversarially mislabeled.