L0-Regularized Quadratic Surface Support Vector Machines

TL;DR

This paper introduces sparse L0-regularized quadratic surface SVMs that improve interpretability and generalization by enforcing sparsity, with an efficient algorithm and strong empirical results on benchmark and credit datasets.

Contribution

The paper proposes a novel sparse quadratic SVM model with an efficient penalty decomposition algorithm that guarantees optimality conditions and demonstrates competitive performance.

Findings

The sparse QSVM achieves comparable accuracy to standard SVMs.

The method produces highly sparse models, enhancing interpretability.

Strong results on credit scoring datasets show practical applicability.

Abstract

Kernel-free quadratic surface support vector machines (QSVM) have recently gained traction due to their flexibility in modeling nonlinear decision boundaries without relying on kernel functions. However, the introduction of a full quadratic classifier significantly increases the number of model parameters, scaling quadratically with data dimensionality, which often leads to overfitting and makes interpretation difficult. To address these challenges, we propose sparse variants of the QSVM by enforcing a cardinality constraint on the model parameters. While enhancing generalization and promoting sparsity, leveraging the $\ell_0$-norm inevitably incurs additional computational complexity. To tackle this, we develop a penalty decomposition algorithm capable of producing solutions that provably satisfy the first-order Lu-Zhang optimality conditions. We show that the subproblems arising within the algorithm either admit closed-form solutions or can be solved efficiently through dual formulations, which contributes to the method's overall effectiveness. Besides, we analyze the convergence behavior of the algorithm under both loss settings. In addition, the numerical experiments on public benchmark datasets indicate that the proposed model is competitive with commonly used SVM variants and produces sparse solutions as expected. Moreover, its strong performance on real-world credit datasets demonstrates its potential for credit scoring applications.