Strong consistency of the AIC, BIC, $C_p$ and KOO methods in high-dimensional multivariate linear regression

TL;DR

This paper investigates the conditions under which classical model selection criteria like AIC, BIC, and $C_p$ are strongly consistent in high-dimensional multivariate linear regression, introduces new methods based on KOO, and demonstrates their improved performance.

Contribution

The paper establishes necessary and sufficient conditions for the strong consistency of AIC, BIC, and $C_p$ in high-dimensional settings and proposes new KOO-based methods with faster convergence.

Findings

BIC is strongly consistent implies AIC is strongly consistent under certain conditions.

Classical criteria may perform poorly in high-dimensional data without proper conditions.

Proposed KOO-based methods are strongly consistent and outperform original methods in simulations.

Abstract

Variable selection is essential for improving inference and interpretation in multivariate linear regression. Although a number of alternative regressor selection criteria have been suggested, the most prominent and widely used are the Akaike information criterion (AIC), Bayesian information criterion (BIC), Mallow's $C_p$, and their modifications. However, for high-dimensional data, experience has shown that the performance of these classical criteria is not always satisfactory. In the present article, we begin by presenting the necessary and sufficient conditions (NSC) for the strong consistency of the high-dimensional AIC, BIC, and $C_p$, based on which we can identify some reasons for their poor performance. Specifically, we show that under certain mild high-dimensional conditions, if the BIC is strongly consistent, then the AIC is strongly consistent, but not vice versa. This result contradicts the classical understanding. In addition, we consider some NSC for the strong consistency of the high-dimensional kick-one-out (KOO) methods introduced by Zhao et al. (1986) and Nishii et al. (1988). Furthermore, we propose two general methods based on the KOO methods and prove their strong consistency. The proposed general methods remove the penalties while simultaneously reducing the conditions for the dimensions and sizes of the regressors. A simulation study supports our consistency conclusions and shows that the convergence rates of the two proposed general KOO methods are much faster than those of the original methods.