A Bernstein-Von Mises Theorem for discrete probability distributions

TL;DR

This paper proves a Bernstein-Von Mises theorem for discrete distributions, showing that under certain conditions, the posterior distribution becomes approximately Gaussian as sample size grows, even with increasing model complexity.

Contribution

It establishes a Bernstein-Von Mises result for high-dimensional discrete models, extending asymptotic normality to settings with increasing model dimension.

Findings

Posterior distribution converges to a Gaussian distribution in variation distance.

The theorem applies to Bayesian estimation of Shannon and Rényi entropies.

Posterior concentration rates are established with respect to Fisher distance.

Abstract

We investigate the asymptotic normality of the posterior distribution in the discrete setting, when model dimension increases with sample size. We consider a probability mass function $\theta_0$ on $\mathbbm{N}\setminus \{0\}$ and a sequence of truncation levels $(k_n)_n$ satisfying $k_n^3\leq n\inf_{i\leq k_n}\theta_0(i).$ Let $\hat{\theta}$ denote the maximum likelihood estimate of $(\theta_0(i))_{i\leq k_n}$ and let $\Delta_n(\theta_0)$ denote the $k_n$-dimensional vector which $i$-th coordinate is defined by \sqrt{n} (\hat{\theta}_n(i)-\theta_0(i)) for $1\leq i\leq k_n.$ We check that under mild conditions on $\theta_0$ and on the sequence of prior probabilities on the $k_n$-dimensional simplices, after centering and rescaling, the variation distance between the posterior distribution recentered around $\hat{\theta}_n$ and rescaled by $\sqrt{n}$ and the $k_n$-dimensional Gaussian distribution $\mathcal{N}(\Delta_n(\theta_0),I^{-1}(\theta_0))$ converges in probability to $0.$ This theorem can be used to prove the asymptotic normality of Bayesian estimators of Shannon and R\'{e}nyi entropies. The proofs are based on concentration inequalities for centered and non-centered Chi-square (Pearson) statistics. The latter allow to establish posterior concentration rates with respect to Fisher distance rather than with respect to the Hellinger distance as it is commonplace in non-parametric Bayesian statistics.