Cross-validation-based optimal feature selection for linear SVM classification

TL;DR

This paper develops a cross-validation-based feature selection method for linear SVM classification, reformulating the problem as a mixed-integer optimization for efficient solution and demonstrating superior performance in experiments.

Contribution

It introduces a novel framework that extends cross-validation feature selection to SVMs using bilevel optimization and LS-SVM reformulation, enabling practical solution.

Findings

The proposed method outperforms L1-regularization and recursive feature elimination in accuracy.

It achieves better feature selection accuracy in simulation experiments.

The framework is computationally feasible with standard optimization software.

Abstract

This paper addresses feature subset selection for Support Vector Machines (SVMs) based on the cross-validation criterion. Unlike statistical criteria such as the Akaike information criterion (AIC) and the Bayesian information criterion (BIC), cross-validation requires only the mild assumption that samples are independently and identically distributed (i.i.d.). For this reason, the cross-validation criterion is expected to work well across a wide range of prediction problems, and it has already demonstrated its usefulness as a feature subset selection method for regression. The objective of this paper is to extend the framework of best feature subset selection via the cross-validation criterion to SVM classification problems. This subset-selection problem can be formulated as a bilevel mixed-integer optimization problem. Because bilevel optimization problems are generally hard to solve, we introduce the Least Squares Support Vector Machine (LS-SVM), whose optimality conditions admit a closed-form expression, and reduce the problem to a single-level mixed-integer optimization problem. This reformulation allows us to solve the problem using standard optimization software. We evaluate the proposed framework through simulation experiments that compare it with a regularization-based method (L1-regularization), a sequential search method (recursive feature elimination), and mixed-integer optimization (MIO) based on statistical criteria. The results show that the proposed framework achieves favorable performance both in classification accuracy and feature selection accuracy.