Robust and consistent model evaluation criteria in high-dimensional regression

TL;DR

This paper introduces new robust model evaluation criteria for high-dimensional linear regression that improve variable selection accuracy and robustness against outliers, outperforming traditional methods like AIC and BIC.

Contribution

It proposes divergence-based criteria with a quasi-Bayesian approach that ensure robustness and selection consistency in high-dimensional settings.

Findings

Proposed criteria outperform traditional methods in simulations.

Criteria maintain robustness against outliers.

Achieve consistent variable selection in high-dimensional models.

Abstract

Most of the regularization methods such as the LASSO have one (or more) regularization parameter(s), and to select the value of the regularization parameter is essentially equal to select a model. Thus, to obtain a model suitable for the data and phenomenon, we need to determine an adequate value of the regularization parameter. Regarding the determination of the regularization parameter in the linear regression model, we often apply the information criteria like the AIC and BIC, however, it has been pointed out that these criteria are sensitive to outliers and tend not to perform well in high-dimensional settings. Outliers generally have a negative effect on not only estimation but also model selection, consequently, it is important to employ a selection method with robustness against outliers. In addition, when the number of explanatory variables is quite large, most conventional criteria are prone to select unnecessary explanatory variables. In this paper, we propose model evaluation criteria based on the statistical divergence with excellence in robustness in both of parametric estimation and model selection, by applying the quasi-Bayesian procedure. Our proposed criteria achieve the selection consistency even in high-dimensional settings due to precise approximation, simultaneously with robustness. We also investigate the conditions for establishing robustness and consistency, and provide an appropriate example of the divergence and penalty term that can achieve the desirable properties. We finally report the results of some numerical examples to verify that the proposed criteria perform robust and consistent variable selection compared with the conventional selection methods.