Consistency of Bayesian procedures for variable selection

TL;DR

This paper demonstrates that Bayesian variable selection procedures using intrinsic priors are consistent across all normal linear models and avoid Lindley's paradox, with asymptotics similar to BIC.

Contribution

It extends Bayesian consistency results from pairwise nested models to the entire class of normal linear models using intrinsic priors.

Findings

Bayesian procedures with intrinsic priors are consistent for all normal linear models.

Asymptotics of Bayes factors with intrinsic priors match BIC.

Intrinsic priors prevent Lindley's paradox in variable selection.

Abstract

It has long been known that for the comparison of pairwise nested models, a decision based on the Bayes factor produces a consistent model selector (in the frequentist sense). Here we go beyond the usual consistency for nested pairwise models, and show that for a wide class of prior distributions, including intrinsic priors, the corresponding Bayesian procedure for variable selection in normal regression is consistent in the entire class of normal linear models. We find that the asymptotics of the Bayes factors for intrinsic priors are equivalent to those of the Schwarz (BIC) criterion. Also, recall that the Jeffreys--Lindley paradox refers to the well-known fact that a point null hypothesis on the normal mean parameter is always accepted when the variance of the conjugate prior goes to infinity. This implies that some limiting forms of proper prior distributions are not necessarily suitable for testing problems. Intrinsic priors are limits of proper prior distributions, and for finite sample sizes they have been proved to behave extremely well for variable selection in regression; a consequence of our results is that for intrinsic priors Lindley's paradox does not arise.