Nonparametric Bayesian model selection and averaging

TL;DR

This paper develops a hierarchical Bayesian framework for nonparametric density estimation that adapts to the true density's smoothness and effectively selects the optimal model, with proven convergence rates.

Contribution

It introduces a general theorem on posterior contraction rates for hierarchical nonparametric Bayesian models, demonstrating adaptive estimation and model selection capabilities.

Findings

Posterior distribution adapts to the smoothness of the true density.

Posterior effectively selects the optimal or smaller models.

Convergence rates depend on prior specifications but show robustness.

Abstract

We consider nonparametric Bayesian estimation of a probability density $p$ based on a random sample of size $n$ from this density using a hierarchical prior. The prior consists, for instance, of prior weights on the regularity of the unknown density combined with priors that are appropriate given that the density has this regularity. More generally, the hierarchy consists of prior weights on an abstract model index and a prior on a density model for each model index. We present a general theorem on the rate of contraction of the resulting posterior distribution as $n\to \infty$, which gives conditions under which the rate of contraction is the one attached to the model that best approximates the true density of the observations. This shows that, for instance, the posterior distribution can adapt to the smoothness of the underlying density. We also study the posterior distribution of the model index, and find that under the same conditions the posterior distribution gives negligible weight to models that are bigger than the optimal one, and thus selects the optimal model or smaller models that also approximate the true density well. We apply these result to log spline density models, where we show that the prior weights on the regularity index interact with the priors on the models, making the exact rates depend in a complicated way on the priors, but also that the rate is fairly robust to specification of the prior weights.