Max-Margin Stacking and Sparse Regularization for Linear Classifier Combination and Selection

TL;DR

This paper explores advanced stacking methods for classifier ensemble combination, introducing sparse regularization techniques that improve classifier selection and can enhance accuracy or reduce complexity.

Contribution

It proposes a novel regularized empirical risk minimization approach with hinge loss and group sparsity for classifier combination and selection in stacking.

Findings

Sparse regularization reduces the number of classifiers without losing accuracy.

Regularized hinge loss improves ensemble performance.

Sparse methods can even increase accuracy in less diverse ensembles.

Abstract

The main principle of stacked generalization (or Stacking) is using a second-level generalizer to combine the outputs of base classifiers in an ensemble. In this paper, we investigate different combination types under the stacking framework; namely weighted sum (WS), class-dependent weighted sum (CWS) and linear stacked generalization (LSG). For learning the weights, we propose using regularized empirical risk minimization with the hinge loss. In addition, we propose using group sparsity for regularization to facilitate classifier selection. We performed experiments using two different ensemble setups with differing diversities on 8 real-world datasets. Results show the power of regularized learning with the hinge loss function. Using sparse regularization, we are able to reduce the number of selected classifiers of the diverse ensemble without sacrificing accuracy. With the non-diverse ensembles, we even gain accuracy on average by using sparse regularization.