A Majorization Penalty Method for SVM with Sparse Constraint

TL;DR

This paper introduces a novel SVM model with a sparse constraint that explicitly limits misclassified samples, solved via a majorization penalty method and conjugate-gradient, showing strong empirical performance.

Contribution

It proposes a new SVM formulation with a sparse constraint for explicit misclassification control, solved using a majorization penalty approach and CG method.

Findings

The proposed method effectively limits misclassified samples.

Numerical results demonstrate superior performance.

The approach outperforms existing SVM models in experiments.

Abstract

Support vector machine is an important and fundamental technique in machine learning. Soft-margin SVM models have stronger generalization performance compared with the hard-margin SVM. Most existing works use the hinge-loss function which can be regarded as an upper bound of the 0-1 loss function. However, it can not explicitly limit the number of misclassified samples. In this paper, we use the idea of soft-margin SVM and propose a new SVM model with a sparse constraint. Our model can strictly limit the number of misclassified samples, expressing the soft-margin constraint as a sparse constraint. By constructing a majorization function, a majorization penalty method can be used to solve the sparse-constrained optimization problem. We apply Conjugate-Gradient (CG) method to solve the resulting subproblem. Extensive numerical results demonstrate the impressive performance of the proposed majorization penalty method.