Subsampled Ensemble Can Improve Generalization Tail Exponentially

TL;DR

This paper introduces a novel ensemble method that selects the most frequently generated models from subsamples, exponentially improving the tail decay of excess risk even with slow base learners, enhancing out-of-sample performance.

Contribution

It presents a new perspective on ensembling that achieves exponential tail decay of excess risk through model frequency, surpassing traditional variance reduction.

Findings

Exponential tail decay achieved with subsampled ensemble.

Improved out-of-sample performance on heavy-tailed data.

Applicable to base learners with slow decay rates.

Abstract

Ensemble learning is a popular technique to improve the accuracy of machine learning models. It traditionally hinges on the rationale that aggregating multiple weak models can lead to better models with lower variance and hence higher stability, especially for discontinuous base learners. In this paper, we provide a new perspective on ensembling. By selecting the most frequently generated model from the base learner when repeatedly applied to subsamples, we can attain exponentially decaying tails for the excess risk, even if the base learner suffers from slow (i.e., polynomial) decay rates. This tail enhancement power of ensembling applies to base learners that have reasonable predictive power to begin with and is stronger than variance reduction in the sense of exhibiting rate improvement. We demonstrate how our ensemble methods can substantially improve out-of-sample performances in a range of numerical examples involving heavy-tailed data or intrinsically slow rates.