Optimal Model Averaging of Support Vector Machines in Diverging Model Spaces

TL;DR

This paper introduces a new frequentist model averaging method for support vector machines that optimally combines models via cross-validation, especially effective in high-dimensional settings, and demonstrates its asymptotic optimality.

Contribution

It proposes the first frequentist model averaging procedure for SVMs that adapts to diverging covariate spaces and avoids complex tuning procedures.

Findings

Asymptotic optimality of the proposed method is established.

The method performs well in empirical studies, outperforming traditional model selection.

Convergence rates provide new insights into model averaging in high-dimensional contexts.

Abstract

Support vector machine (SVM) is a powerful classification method that has achieved great success in many fields. Since its performance can be seriously impaired by redundant covariates, model selection techniques are widely used for SVM with high dimensional covariates. As an alternative to model selection, significant progress has been made in the area of model averaging in the past decades. Yet no frequentist model averaging method was considered for SVM. This work aims to fill the gap and to propose a frequentist model averaging procedure for SVM which selects the optimal weight by cross validation. Even when the number of covariates diverges at an exponential rate of the sample size, we show asymptotic optimality of the proposed method in the sense that the ratio of its hinge loss to the lowest possible loss converges to one. We also derive the convergence rate which provides more insights to model averaging. Compared to model selection methods of SVM which require a tedious but critical task of tuning parameter selection, the model averaging method avoids the task and shows promising performances in the empirical studies.