Regularization in regression: comparing Bayesian and frequentist methods in a poorly informative situation

TL;DR

This paper compares Bayesian and frequentist regularization methods in low-informative, high-dimensional regression settings using simulated and real data, highlighting the advantages of Bayesian approaches in variable selection and prediction accuracy.

Contribution

It introduces new noninformative Bayesian variable selection methods based on Zellner's g-priors and demonstrates their superiority over frequentist methods in predictive performance and variable selection.

Findings

Bayesian methods yield smaller prediction errors.

Bayesian approaches select more relevant variables.

Bayesian regularization outperforms frequentist methods in simulations.

Abstract

Using a collection of simulated an real benchmarks, we compare Bayesian and frequentist regularization approaches under a low informative constraint when the number of variables is almost equal to the number of observations on simulated and real datasets. This comparison includes new global noninformative approaches for Bayesian variable selection built on Zellner's g-priors that are similar to Liang et al. (2008). The interest of those calibration-free proposals is discussed. The numerical experiments we present highlight the appeal of Bayesian regularization methods, when compared with non-Bayesian alternatives. They dominate frequentist methods in the sense that they provide smaller prediction errors while selecting the most relevant variables in a parsimonious way.