Model Averaging for Support Vector Machine by Cross-Validation

TL;DR

This paper introduces a model averaging approach for support vector machines that combines estimates from multiple models, improving predictive accuracy and stability over traditional model selection methods.

Contribution

It proposes a novel weighting scheme for SVM model averaging, with theoretical guarantees and a feature screening step to reduce computational complexity.

Findings

Achieves the smallest asymptotic hinge risk among feasible model combinations.

Demonstrates improved accuracy over existing methods in real data and simulations.

Provides a computationally efficient approach through feature screening.

Abstract

Support vector machine (SVM) is a well-known statistical technique for classification problems in machine learning and other fields. An important question for SVM is the selection of covariates (or features) for the model. Many studies have considered model selection methods. As is well-known, selecting one winning model over others can entail considerable instability in predictive performance due to model selection uncertainties. This paper advocates model averaging as an alternative approach, where estimates obtained from different models are combined in a weighted average. We propose a model weighting scheme and provide the theoretical underpinning for the proposed method. In particular, we prove that our proposed method yields a model average estimator that achieves the smallest hinge risk among all feasible combinations asymptotically. To remedy the computational burden due to a large number of feasible models, we propose a screening step to eliminate the uninformative features before combining the models. Results from real data applications and a simulation study show that the proposed method generally yields more accurate estimates than existing methods.