Choosing the Right Regularizer for Applied ML: Simulation Benchmarks of Popular Scikit-learn Regularization Frameworks

TL;DR

This paper empirically compares popular regularization techniques in scikit-learn across extensive simulations, providing practical guidance for selecting the best method based on data characteristics.

Contribution

It offers a comprehensive benchmark of Ridge, Lasso, ElasticNet, and Post-Lasso OLS, highlighting their performance differences and robustness under various conditions.

Findings

Ridge, Lasso, and ElasticNet perform similarly with sufficient data.

Lasso's recall drops significantly under multicollinearity and low SNR.

ElasticNet maintains high recall where Lasso fails, especially at high condition numbers.

Abstract

This study surveys the historical development of regularization, tracing its evolution from stepwise regression in the 1960s to recent advancements in formal error control, structured penalties for non-independent features, Bayesian methods, and l0-based regularization (among other techniques). We empirically evaluate the performance of four canonical frameworks -- Ridge, Lasso, ElasticNet, and Post-Lasso OLS -- across 134,400 simulations spanning a 7-dimensional manifold grounded in eight production-grade machine learning models. Our findings demonstrate that for prediction accuracy when the sample-to-feature ratio is sufficient (n/p >= 78), Ridge, Lasso, and ElasticNet are nearly interchangeable. However, we find that Lasso recall is highly fragile under multicollinearity; at high condition numbers (kappa) and low SNR, Lasso recall collapses to 0.18 while ElasticNet maintains 0.93. Consequently, we advise practitioners against using Lasso or Post-Lasso OLS at high kappa with small sample sizes. The analysis concludes with an objective-driven decision guide to assist machine learning engineers in selecting the optimal scikit-learn-supported framework based on observable feature space attributes.