Algorithms for Sparse Support Vector Machines

TL;DR

This paper introduces a novel approach for sparse support vector machines using set constraints and proximal distance algorithms, leading to better sparsity and classification accuracy compared to traditional $ ext{l}_1$ penalty methods.

Contribution

It proposes an alternative to $ ext{l}_1$ penalties by employing sparse-set constraints and proximal distance algorithms for improved feature selection in support vector machines.

Findings

Achieves better sparsity without sacrificing classification power.

Demonstrates effectiveness through simulated and real data examples.

Provides algorithms that outperform traditional $ ext{l}_1$ penalty methods.

Abstract

Many problems in classification involve huge numbers of irrelevant features. Model selection reveals the crucial features, reduces the dimensionality of feature space, and improves model interpretation. In the support vector machine literature, model selection is achieved by $\ell_1$ penalties. These convex relaxations seriously bias parameter estimates toward 0 and tend to admit too many irrelevant features. The current paper presents an alternative that replaces penalties by sparse-set constraints. Penalties still appear, but serve a different purpose. The proximal distance principle takes a loss function $L(\boldsymbol{\beta})$ and adds the penalty $\frac{\rho}{2}\mathrm{dist}(\boldsymbol{\beta}, S_k)^2$ capturing the squared Euclidean distance of the parameter vector $\boldsymbol{\beta}$ to the sparsity set $S_k$ where at most $k$ components of $\boldsymbol{\beta}$ are nonzero. If $\boldsymbol{\beta}_\rho$ represents the minimum of the objective $f_\rho(\boldsymbol{\beta})=L(\boldsymbol{\beta})+\frac{\rho}{2}\mathrm{dist}(\boldsymbol{\beta}, S_k)^2$, then $\boldsymbol{\beta}_\rho$ tends to the constrained minimum of $L(\boldsymbol{\beta})$ over $S_k$ as $\rho$ tends to $\infty$. We derive two closely related algorithms to carry out this strategy. Our simulated and real examples vividly demonstrate how the algorithms achieve much better sparsity without loss of classification power.