On the proliferation of support vectors in high dimensions

TL;DR

This paper investigates why support vector machines can still perform well in high-dimensional spaces despite having many support vectors, by establishing new theoretical equivalences and conditions.

Contribution

It introduces new deterministic equivalences that explain support vector proliferation and broadens the understanding of high-dimensional SVM behavior.

Findings

Support vector proliferation occurs under broader conditions than previously known.

High-dimensional SVMs can generalize well even when all training points are support vectors.

The paper provides a nearly matching converse result to the support vector proliferation phenomenon.

Abstract

The support vector machine (SVM) is a well-established classification method whose name refers to the particular training examples, called support vectors, that determine the maximum margin separating hyperplane. The SVM classifier is known to enjoy good generalization properties when the number of support vectors is small compared to the number of training examples. However, recent research has shown that in sufficiently high-dimensional linear classification problems, the SVM can generalize well despite a proliferation of support vectors where all training examples are support vectors. In this paper, we identify new deterministic equivalences for this phenomenon of support vector proliferation, and use them to (1) substantially broaden the conditions under which the phenomenon occurs in high-dimensional settings, and (2) prove a nearly matching converse result.