A General Framework for Bayes Structured Linear Models

TL;DR

This paper introduces a unified Bayesian framework for high-dimensional and nonparametric structured linear models, providing theoretical guarantees for posterior contraction in complex settings like stochastic block models and graphon estimation.

Contribution

It proposes a two-step model selection prior that enables deriving optimal posterior contraction results across various high-dimensional and nonparametric models.

Findings

Proves a general theorem for posterior contraction in structured linear models.

Derives new optimal contraction results for complex models like stochastic block models.

Re-derives and improves results for sparse linear regression and nonparametric aggregation.

Abstract

High dimensional statistics deals with the challenge of extracting structured information from complex model settings. Compared with the growing number of frequentist methodologies, there are rather few theoretically optimal Bayes methods that can deal with very general high dimensional models. In contrast, Bayes methods have been extensively studied in various nonparametric settings and rate optimal posterior contraction results have been established. This paper provides a unified approach to both Bayes high dimensional statistics and Bayes nonparametrics in a general framework of structured linear models. With the proposed two-step model selection prior, we prove a general theorem of posterior contraction under an abstract setting. The main theorem can be used to derive new results on optimal posterior contraction under many complex model settings including stochastic block model, graphon estimation and dictionary learning. It can also be used to re-derive optimal posterior contraction for problems such as sparse linear regression and nonparametric aggregation, which improve upon previous Bayes results for these problems. The key of the success lies in the proposed two-step prior distribution. The prior on the parameters is an elliptical Laplace distribution that is capable to model signals with large magnitude, and the prior on the models involves an important correction factor that compensates the effect of the normalizing constant of the elliptical Laplace distribution.