No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions

TL;DR

This paper analyzes the trade-offs between single large models and ensembles of smaller models in random-feature ridge regression, showing that larger models generally perform better unless specific conditions favor ensembles, especially in overparameterized regimes.

Contribution

It provides theoretical scaling laws and optimality conditions for ensembles versus single models in high-dimensional ridge regression, highlighting when ensembles can be near-optimal.

Findings

Single large models outperform ensembles with fixed total parameters.

Overparameterized ensembles can achieve near-optimal performance.

Scaling laws depend on kernel and task eigenstructure.

Abstract

Given a fixed budget for total model size, one must choose between training a single large model or combining the predictions of multiple smaller models. We investigate this trade-off for ensembles of random-feature ridge regression models in both the overparameterized and underparameterized regimes. Using deterministic equivalent risk estimates, we prove that when a fixed number of parameters is distributed among $K$ independently trained models, the ridge-optimized test risk increases with $K$. Consequently, a single large model achieves optimal performance. We then ask when ensembles can achieve \textit{near}-optimal performance. In the overparameterized regime, we show that, to leading order, the test error depends on ensemble size and model size only through the total feature count, so that overparameterized ensembles consistently achieve near-optimal performance. To understand underparameterized ensembles, we derive scaling laws for the test risk as a function of total parameter count when the ensemble size and parameters per ensemble member are jointly scaled according to a ``growth exponent'' $\ell$. While the optimal error scaling is always achieved by increasing model size with a fixed ensemble size, our analysis identifies conditions on the kernel and task eigenstructure under which near-optimal scaling laws can be obtained by joint scaling of ensemble size and model size.