Sparse SVM for Sufficient Data Reduction

TL;DR

This paper introduces a Newton-type method for sparse SVMs that effectively reduces the number of support vectors and computational time for large datasets, enhancing efficiency in kernel-based SVM applications.

Contribution

The paper develops a novel Newton-type optimization algorithm for sparsity constrained kernel SVMs with one-step convergence near stationary points.

Findings

Achieves significantly fewer support vectors in large datasets.

Demonstrates shorter computational times compared to existing solvers.

Performs exceptionally well in numerical experiments.

Abstract

Kernel-based methods for support vector machines (SVM) have shown highly advantageous performance in various applications. However, they may incur prohibitive computational costs for large-scale sample datasets. Therefore, data reduction (reducing the number of support vectors) appears to be necessary, which gives rise to the topic of the sparse SVM. Motivated by this problem, the sparsity constrained kernel SVM optimization has been considered in this paper in order to control the number of support vectors. Based on the established optimality conditions associated with the stationary equations, a Newton-type method is developed to handle the sparsity constrained optimization. This method is found to enjoy the one-step convergence property if the starting point is chosen to be close to a local region of a stationary point, thereby leading to a super-high computational speed. Numerical comparisons with several powerful solvers demonstrate that the proposed method performs exceptionally well, particularly for large-scale datasets in terms of a much lower number of support vectors and shorter computational time.