Beyond prediction: A framework for inference with variational approximations in mixture models

TL;DR

This paper develops a theoretical framework for inference with variational approximations in mixture models, establishing conditions for estimator consistency and asymptotic normality, and proposes methodological improvements.

Contribution

It connects variational estimators to profile M-estimation, providing regularity conditions and introducing new methods for improved inference.

Findings

Establishes regularity conditions for variational estimators' consistency and normality.

Proposes methods for covariance estimation, efficiency correction, and empirical consistency assessment.

Validates theoretical results through simulations and real data analysis.

Abstract

Variational inference is a popular method for estimating model parameters and conditional distributions in hierarchical and mixed models, which arise frequently in many settings in the health, social, and biological sciences. Variational inference in a frequentist context works by approximating intractable conditional distributions with a tractable family and optimizing the resulting lower bound on the log-likelihood. The variational objective function is typically less computationally intensive to optimize than the true likelihood, enabling scientists to fit rich models even with extremely large datasets. Despite widespread use, little is known about the general theoretical properties of estimators arising from variational approximations to the log-likelihood, which hinders their use in inferential statistics. In this paper we connect such estimators to profile M-estimation, which enables us to provide regularity conditions for consistency and asymptotic normality of variational estimators. Our theory also motivates three methodological improvements to variational inference: estimation of the asymptotic model-robust covariance matrix, a one-step correction that improves estimator efficiency, and an empirical assessment of consistency. We evaluate the proposed results using simulation studies and data on marijuana use from the National Longitudinal Study of Youth.