Support vector machines and Radon's theorem

TL;DR

This paper links the configurations of support vectors in SVMs to Radon's theorem, providing geometric insights, stability conditions, and simulation results on support vector counts and arrangements.

Contribution

It introduces a novel connection between support vector configurations and Radon's theorem, offering new geometric understanding and stability analysis of SVM support vectors.

Findings

Support vectors intersect in exactly one point under certain conditions.

Support vectors are stable under perturbations.

Number of support vectors ranges from 2 to n+1, depending on data.

Abstract

A support vector machine (SVM) is an algorithm that finds a hyperplane which optimally separates labeled data points in $\mathbb{R}^n$ into positive and negative classes. The data points on the margin of this separating hyperplane are called support vectors. We connect the possible configurations of support vectors to Radon's theorem, which provides guarantees for when a set of points can be divided into two classes (positive and negative) whose convex hulls intersect. If the convex hulls of the positive and negative support vectors are projected onto a separating hyperplane, then the projections intersect if and only if the hyperplane is optimal. Further, with a particular type of general position, we show that (a) the projected convex hulls of the support vectors intersect in exactly one point, (b) the support vectors are stable under perturbation, (c) there are at most $n+1$ support vectors, and (d) every number of support vectors from 2 up to $n+1$ is possible. Finally, we perform computer simulations studying the expected number of support vectors, and their configurations, for randomly generated data. We observe that as the distance between classes of points increases for this type of randomly generated data, configurations with fewer support vectors become more likely.