Robustness and Regularization of Support Vector Machines

TL;DR

This paper reveals that regularized support vector machines are equivalent to a robust optimization formulation, providing insights into their robustness, generalization, and potential for noise protection in classification algorithms.

Contribution

It establishes a novel equivalence between regularization in SVMs and robust optimization, offering new perspectives for algorithms and analysis.

Findings

Robust optimization formulation of SVMs explains their success.

Robustness interpretation provides a new proof of SVMs' consistency.

Supports the idea that regularized SVMs generalize well due to robustness.

Abstract

We consider regularized support vector machines (SVMs) and show that they are precisely equivalent to a new robust optimization formulation. We show that this equivalence of robust optimization and regularization has implications for both algorithms, and analysis. In terms of algorithms, the equivalence suggests more general SVM-like algorithms for classification that explicitly build in protection to noise, and at the same time control overfitting. On the analysis front, the equivalence of robustness and regularization, provides a robust optimization interpretation for the success of regularized SVMs. We use the this new robustness interpretation of SVMs to give a new proof of consistency of (kernelized) SVMs, thus establishing robustness as the reason regularized SVMs generalize well.