Statistical Mechanics of Support Vector Networks

TL;DR

This paper applies statistical physics methods to analyze support vector machines, revealing their generalization behavior, overfitting tendencies, and how input distribution gaps influence performance.

Contribution

It introduces a statistical mechanics framework to study SVMs, providing insights into their generalization error and effects of input distribution characteristics.

Findings

Generalization error saturates on a plateau for nonlinear rules

SVMs overfit weakly on simple rules

Input distribution gaps significantly improve SVM performance

Abstract

Using methods of Statistical Physics, we investigate the generalization performance of support vector machines (SVMs), which have been recently introduced as a general alternative to neural networks. For nonlinear classification rules, the generalization error saturates on a plateau, when the number of examples is too small to properly estimate the coefficients of the nonlinear part. When trained on simple rules, we find that SVMs overfit only weakly. The performance of SVMs is strongly enhanced, when the distribution of the inputs has a gap in feature space.