Error Reduction from Stacked Regressions

TL;DR

This paper introduces a novel approach to stacking regressions by learning combination weights through regularized empirical risk minimization, resulting in improved predictive accuracy especially in low signal-to-noise scenarios.

Contribution

It demonstrates that under certain conditions, the proposed stacking method achieves strictly lower population risk than any single estimator, with computational efficiency comparable to isotonic regression.

Findings

Stacked estimator outperforms individual estimators in risk reduction.

The method is particularly effective when the signal-to-noise ratio is low.

Computational complexity is similar to that of isotonic regression.

Abstract

Stacking regressions is an ensemble technique that forms linear combinations of different regression estimators to enhance predictive accuracy. The conventional approach uses cross-validation data to generate predictions from the constituent estimators, and least-squares with nonnegativity constraints to learn the combination weights. In this paper, we learn these weights analogously by minimizing a regularized version of the empirical risk subject to a nonnegativity constraint. When the constituent estimators are linear least-squares projections onto nested subspaces separated by at least three dimensions, we show that thanks to an adaptive shrinkage effect, the resulting stacked estimator has strictly smaller population risk than best single estimator among them, with more significant gains when the signal-to-noise ratio is small. Here "best" refers to an estimator that minimizes a model selection criterion such as AIC or BIC. In other words, in this setting, the best single estimator is inadmissible. Because the optimization problem can be reformulated as isotonic regression, the stacked estimator requires the same order of computation as the best single estimator, making it an attractive alternative in terms of both performance and implementation.