Combinatorial Approaches for Embedded Feature Selection in Nonlinear SVMs

TL;DR

This paper introduces a novel approach to embedded feature selection in nonlinear SVMs by embedding a hard cardinality constraint directly into the dual formulation, using a decomposition framework and submodular optimization for improved solutions.

Contribution

It proposes the first hard-constraint embedded feature selection method for nonlinear SVMs in the dual form, with scalable algorithms and a decomposition framework for enhanced performance.

Findings

Algorithms outperform standard methods in numerical experiments.

The binary subproblem reduces to submodular maximization, enabling scalable solutions.

The approach guarantees strict control over the number of selected features.

Abstract

Embedded Feature Selection (FS) is a classical approach for interpretable machine learning, aiming to identify the most relevant features of a dataset while simultaneously training the model. We consider an approach based on a hard cardinality constraint for nonlinear SVMs. To the best of our knowledge, hard-constraint approaches have been proposed only for the primal formulation of linear SVMs. In contrast, we embed a hard cardinality constraint directly into the dual of a nonlinear SVM, guaranteeing strict control over the number of selected features while still leveraging kernelization. We formulate the problem as a Mixed-Integer Nonlinear Programming (MINLP) model. As a first contribution, we propose a local search metaheuristic applicable to general nonlinear kernels. Our second and main contribution is a decomposition framework that alternates optimization between two subproblems: one involving only continuous variables and the other involving only binary variables. For polynomial kernels, we show that the binary subproblem reduces to a submodular function maximization under a cardinality constraint, enabling the use of scalable submodular maximization algorithms within the alternating optimization process. Numerical experiments demonstrate that our algorithms significantly outperform standard methods for solving the proposed MINLPs, providing more effective solutions to the addressed feature selection problem.