Bayesian model choice and information criteria in sparse generalized linear models

TL;DR

This paper demonstrates that Bayesian model selection in high-dimensional sparse generalized linear models is asymptotically equivalent to using an extended Bayesian information criterion, ensuring consistent model selection.

Contribution

It establishes the asymptotic equivalence between Bayesian model selection with certain priors and the extended Bayesian information criterion in high-dimensional sparse settings.

Findings

Bayesian model selection is asymptotically equivalent to EBIC.

The smallest true model is consistently selected.

EBIC performs well on real data for regression and graphical models.

Abstract

We consider Bayesian model selection in generalized linear models that are high-dimensional, with the number of covariates p being large relative to the sample size n, but sparse in that the number of active covariates is small compared to p. Treating the covariates as random and adopting an asymptotic scenario in which p increases with n, we show that Bayesian model selection using certain priors on the set of models is asymptotically equivalent to selecting a model using an extended Bayesian information criterion. Moreover, we prove that the smallest true model is selected by either of these methods with probability tending to one. Having addressed random covariates, we are also able to give a consistency result for pseudo-likelihood approaches to high-dimensional sparse graphical modeling. Experiments on real data demonstrate good performance of the extended Bayesian information criterion for regression and for graphical models.