Asymptotic Consistency and Generalization in Hybrid Models of Regularized Selection and Nonlinear Learning

TL;DR

This paper evaluates regularized, black-box, and hybrid models for variable selection and prediction in high-dimensional noisy data, finding hybrid models offer a good balance of accuracy, interpretability, and robustness.

Contribution

It introduces and assesses hybrid models combining regularization and nonlinear algorithms, demonstrating their advantages over pure approaches in variable selection and prediction.

Findings

Black-box models have higher predictive accuracy but lower interpretability.

Hybrid models balance accuracy and interpretability effectively.

Hybrid models are more robust and better at identifying relevant variables as sample size grows.

Abstract

This study explores how different types of supervised models perform in the task of predicting and selecting relevant variables in high-dimensional contexts, especially when the data is very noisy. We analyzed three approaches: regularized models (such as Lasso, Ridge, and Elastic Net), black-box models (such as Random Forest, XGBoost, LightGBM, CatBoost, and H2O GBM), and hybrid models that combine both approaches: regularization with nonlinear algorithms. Based on simulations inspired by the Friedman equation, we evaluated 23 models using three complementary metrics: RMSE, Jaccard index, and recall rate. The results reveal that, although black-box models excel in predictive accuracy, they lack interpretability and simplicity, essential factors in many real-world contexts. Regularized models, on the other hand, proved to be more sensitive to an excess of irrelevant variables. In this scenario, hybrid models stood out for their balance: they maintain good predictive performance, identify relevant variables more consistently, and offer greater robustness, especially as the sample size increases. Therefore, we recommend using this hybrid framework in market applications, where it is essential that the results make sense in a practical context and support decisions with confidence.