Concentration of posterior probabilities and normalized L0 criteria

TL;DR

This paper analyzes how Bayesian and $L_0$ model selection methods concentrate on optimal models in high-dimensional regression, providing theoretical bounds on error probabilities and insights into the effects of sparsity and model misspecification.

Contribution

It introduces a framework to study concentration of posterior probabilities and $L_0$ criteria, validating their use for controlling error rates in high-dimensional model selection.

Findings

Posterior probabilities and $L_0$ criteria can effectively concentrate on the true model.

Less sparse models can achieve consistency and better finite-sample performance.

New bounds for misspecified models and improved rates for non-local priors are established.

Abstract

We study frequentist properties of Bayesian and $L_0$ model selection, with a focus on (potentially non-linear) high-dimensional regression. We propose a construction to study how posterior probabilities and normalized $L_0$ criteria concentrate on the (Kullback-Leibler) optimal model and other subsets of the model space. When such concentration occurs, one also bounds the frequentist probabilities of selecting the correct model, type I and type II errors. These results hold generally, and help validate the use of posterior probabilities and $L_0$ criteria to control frequentist error probabilities associated to model selection and hypothesis tests. Regarding regression, we help understand the effect of the sparsity imposed by the prior or the $L_0$ penalty, and of problem characteristics such as the sample size, signal-to-noise, dimension and true sparsity. A particular finding is that one may use less sparse formulations than would be asymptotically optimal, but still attain consistency and often also significantly better finite-sample performance. We also prove new results related to misspecifying the mean or covariance structures, and give tighter rates for certain non-local priors than currently available.