Asymptotic Consistency of $\alpha-$R\'enyi-Approximate Posteriors

TL;DR

This paper investigates the asymptotic consistency of $oldsymbol{ extalpha}$-Rényi approximate posteriors, revealing conditions under which these variational Bayesian methods reliably approximate true posteriors, especially for $oldsymbol{ extalpha > 1}$ and in models with latent variables.

Contribution

It provides the first comprehensive analysis of the consistency of $ extalpha$-Rényi variational posteriors, including cases with $ extalpha > 1$, and characterizes the properties of good approximation sequences.

Findings

Consistency holds under specific conditions related to the convergence rate of approximation sequences.

Sequences that converge too quickly cannot be considered good sequences.

The analysis extends to $ extalpha=1$ and models with local latent variables.

Abstract

We study the asymptotic consistency properties of $\alpha$-R\'enyi approximate posteriors, a class of variational Bayesian methods that approximate an intractable Bayesian posterior with a member of a tractable family of distributions, the member chosen to minimize the $\alpha$-R\'enyi divergence from the true posterior. Unique to our work is that we consider settings with $\alpha > 1$, resulting in approximations that upperbound the log-likelihood, and consequently have wider spread than traditional variational approaches that minimize the Kullback-Liebler (KL) divergence from the posterior. Our primary result identifies sufficient conditions under which consistency holds, centering around the existence of a 'good' sequence of distributions in the approximating family that possesses, among other properties, the right rate of convergence to a limit distribution. We further characterize the good sequence by demonstrating that a sequence of distributions that converges too quickly cannot be a good sequence. We also extend our analysis to the setting where $\alpha$ equals one, corresponding to the minimizer of the reverse KL divergence, and to models with local latent variables. We also illustrate the existence of good sequence with a number of examples. Our results complement a growing body of work focused on the frequentist properties of variational Bayesian methods.