Non-asymptotic Analysis of $\ell_1$-norm Support Vector Machines

TL;DR

This paper provides the first non-asymptotic theoretical analysis of $$-norm SVMs, showing they can accurately identify sparse classifiers with a number of samples proportional to sparsity and logarithm of dimension.

Contribution

It introduces the first non-asymptotic performance bounds for $$-SVMs in high-dimensional sparse classification, using concentration of measure techniques.

Findings

Sparse classifier can be recovered with $O(s\,\log d)$ samples.

High probability bounds on the accuracy of $$-SVMs.

Application of concentration inequalities in Banach spaces.

Abstract

Support Vector Machines (SVM) with $\ell_1$ penalty became a standard tool in analysis of highdimensional classification problems with sparsity constraints in many applications including bioinformatics and signal processing. Although SVM have been studied intensively in the literature, this paper has to our knowledge first non-asymptotic results on the performance of $\ell_1$-SVM in identification of sparse classifiers. We show that a $d$-dimensional $s$-sparse classification vector can be (with high probability) well approximated from only $O(s\log(d))$ Gaussian trials. The methods used in the proof include concentration of measure and probability in Banach spaces.