On the Asymptotics of Importance Weighted Variational Inference

TL;DR

This paper provides a theoretical analysis of importance weighted variational inference, establishing its consistency, asymptotic normality, and efficiency as sample sizes grow, and explores different regimes based on importance ratio smoothness.

Contribution

It offers the first rigorous theoretical results on the asymptotic behavior of importance weighted variational inference, including conditions for consistency and efficiency.

Findings

Proves consistency of importance weighted variational inference as data and Monte Carlo samples increase.

Establishes asymptotic normality and efficiency under specific growth conditions.

Identifies regimes based on the smoothness of the importance ratio affecting inference properties.

Abstract

For complex latent variable models, the likelihood function is not available in closed form. In this context, a popular method to perform parameter estimation is Importance Weighted Variational Inference. It essentially maximizes the expectation of the logarithm of an importance sampling estimate of the likelihood with respect to both the latent variable model parameters and the importance distribution parameters, the expectation being itself with respect to the importance samples. Despite its great empirical success in machine learning, a theoretical analysis of the limit properties of the resulting estimates is still lacking. We fill this gap by establishing consistency when both the Monte Carlo and the observed data sample sizes go to infinity simultaneously. We also establish asymptotic normality and efficiency under additional conditions relating the rate of growth between the Monte Carlo and the observed data samples sizes. We distinguish several regimes related to the smoothness of the importance ratio.