Random Subset Averaging

TL;DR

The paper introduces Random Subset Averaging (RSA), a novel ensemble method for high-dimensional, correlated data that adaptively combines models to improve prediction accuracy and stability.

Contribution

RSA is a new ensemble prediction approach that constructs models via random subsets and aggregates them with a data-driven weighting scheme, requiring no prior covariate knowledge.

Findings

RSA achieves lower finite-sample risk bounds under orthogonal design.

RSA outperforms traditional methods in diverse simulation scenarios.

RSA demonstrates practical utility in financial return forecasting.

Abstract

We propose a new ensemble prediction method, Random Subset Averaging (RSA), tailored for settings with many covariates, particularly in the presence of strong correlations. RSA constructs candidate models via binomial random subset strategy and aggregates their predictions through a two-round weighting scheme, resulting in a structure analogous to a two-layer neural network. All tuning parameters are selected via cross-validation, requiring no prior knowledge of covariate relevance. We establish the asymptotic optimality of RSA under general conditions, allowing the first-round weights to be data-dependent, and demonstrate that RSA achieves a lower finite-sample risk bound under orthogonal design. Simulation studies demonstrate that RSA consistently delivers superior and stable predictive performance across a wide range of sample sizes, dimensional settings, sparsity levels and correlation structures, outperforming conventional model selection and ensemble learning methods. An empirical application to financial return forecasting further illustrates its practical utility.