High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and ranking

TL;DR

This empirical study compares various penalized regression methods across numerous simulated and semi-synthetic scenarios to evaluate their effectiveness in prediction, variable selection, and ranking in high-dimensional settings.

Contribution

It provides a comprehensive empirical comparison of popular high-dimensional regression methods, highlighting their strengths and limitations in practical finite-sample scenarios.

Findings

No single method dominates across all scenarios.

Performance varies significantly depending on data characteristics.

Recommendations are provided for method selection based on specific goals and data features.

Abstract

Penalized likelihood approaches are widely used for high-dimensional regression. Although many methods have been proposed and the associated theory is now well-developed, the relative efficacy of different approaches in finite-sample settings, as encountered in practice, remains incompletely understood. There is therefore a need for empirical investigations in this area that can offer practical insight and guidance to users. In this paper we present a large-scale comparison of penalized regression methods. We distinguish between three related goals: prediction, variable selection and variable ranking. Our results span more than 2,300 data-generating scenarios, including both synthetic and semi-synthetic data (real covariates and simulated responses), allowing us to systematically consider the influence of various factors (sample size, dimensionality, sparsity, signal strength and multicollinearity). We consider several widely-used approaches (Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability Selection). We find considerable variation in performance between methods. Our results support a `no panacea' view, with no unambiguous winner across all scenarios or goals, even in this restricted setting where all data align well with the assumptions underlying the methods. The study allows us to make some recommendations as to which approaches may be most (or least) suitable given the goal and some data characteristics. Our empirical results complement existing theory and provide a resource to compare methods across a range of scenarios and metrics.