Fast SVM-based Feature Elimination Utilizing Data Radius, Hard-Margin, Soft-Margin

TL;DR

This paper introduces novel SVM-based feature elimination methods that leverage data radius and margin concepts, improving generalization and classification accuracy on high-dimensional datasets.

Contribution

It proposes new radius-based feature elimination criteria combined with a low-cost soft-margin retraining approach called QP1, outperforming previous methods like MFE-LO.

Findings

Our methods outperform MFE-LO in generalization accuracy.

The proposed criteria achieve lower test error rates on large feature datasets.

Tunable methods show potential for even higher future performance.

Abstract

Margin maximization in the hard-margin sense, proposed as feature elimination criterion by the MFE-LO method, is combined here with data radius utilization to further aim to lower generalization error, as several published bounds and bound-related formulations pertaining to lowering misclassification risk (or error) pertain to radius e.g. product of squared radius and weight vector squared norm. Additionally, we propose additional novel feature elimination criteria that, while instead being in the soft-margin sense, too can utilize data radius, utilizing previously published bound-related formulations for approaching radius for the soft-margin sense, whereby e.g. a focus was on the principle stated therein as "finding a bound whose minima are in a region with small leave-one-out values may be more important than its tightness". These additional criteria we propose combine radius utilization with a novel and computationally low-cost soft-margin light classifier retraining approach we devise named QP1; QP1 is the soft-margin alternative to the hard-margin LO. We correct an error in the MFE-LO description, find MFE-LO achieves the highest generalization accuracy among the previously published margin-based feature elimination (MFE) methods, discuss some limitations of MFE-LO, and find our novel methods herein outperform MFE-LO, attain lower test set classification error rate. On several datasets that each both have a large number of features and fall into the `large features few samples' dataset category, and on datasets with lower (low-to-intermediate) number of features, our novel methods give promising results. Especially, among our methods the tunable ones, that do not employ (the non-tunable) LO approach, can be tuned more aggressively in the future than herein, to aim to demonstrate for them even higher performance than herein.