Normal approximation for the posterior in exponential families

TL;DR

This paper provides explicit, non-asymptotic bounds on the accuracy of normal approximations for posterior distributions in exponential family models, applicable in high-dimensional and non-conjugate prior settings.

Contribution

It introduces a flexible Stein's method-based approach for deriving quantitative bounds on posterior normality, with potential for faster convergence rates and broad applicability.

Findings

Bounds valid for univariate and multivariate posteriors

Conditions for faster O(n^{-1}) convergence rates

Impact of prior and data statistics on approximation quality

Abstract

In this paper, we obtain quantitative, non-asymptotic, and data-dependent \textit{Bernstein-von Mises type} bounds on the normal approximation of the posterior distribution in exponential family models with arbitrary centring and scaling. Our bounds, stated in the total variation and Wasserstein distances, are valid for univariate and multivariate posteriors alike, and do not require a conjugate prior setting. They are obtained through a refined version of Stein's method of comparison of operators that allows for improved dimensional dependence in high-dimensional settings and may also be of interest in other problems. Our approach is rather flexible and, in certain settings, allows for the derivation of bounds with rates of convergence faster than the usual \( O(n^{-1/2}) \) rate (when \( n \) is the sample size). We illustrate our findings on a variety of exponential family distributions, including the Weibull, multinomial, and linear regression with unknown variance. The resulting bounds have an explicit dependence on the prior distribution and on sufficient statistics of the data from the sample, and thus provide insight into how these factors affect the quality of the normal approximation. Insights from our examples include identification of conditions under which faster \( O(n^{-1}) \) convergence rates occur for Bernoulli data, illustrations of how the quality of the normal approximation is influenced by the choice of standardisation, and dimensional dependence in high-dimensional settings.