Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data

TL;DR

This paper introduces Variational Gibbs Inference (VGI), a novel method for estimating statistical model parameters from incomplete data, effectively handling missing data issues in models like VAEs and normalising flows.

Contribution

The paper proposes VGI, a general-purpose variational inference technique for incomplete data, addressing intractability in standard VI for missing data scenarios.

Findings

VGI performs competitively or better than existing methods.

VGI successfully estimates models from incomplete real-world data.

VGI is applicable to models like VAEs and normalising flows.

Abstract

Statistical models are central to machine learning with broad applicability across a range of downstream tasks. The models are controlled by free parameters that are typically estimated from data by maximum-likelihood estimation or approximations thereof. However, when faced with real-world data sets many of the models run into a critical issue: they are formulated in terms of fully-observed data, whereas in practice the data sets are plagued with missing data. The theory of statistical model estimation from incomplete data is conceptually similar to the estimation of latent-variable models, where powerful tools such as variational inference (VI) exist. However, in contrast to standard latent-variable models, parameter estimation with incomplete data often requires estimating exponentially-many conditional distributions of the missing variables, hence making standard VI methods intractable. We address this gap by introducing variational Gibbs inference (VGI), a new general-purpose method to estimate the parameters of statistical models from incomplete data. We validate VGI on a set of synthetic and real-world estimation tasks, estimating important machine learning models such as variational autoencoders and normalising flows from incomplete data. The proposed method, whilst general-purpose, achieves competitive or better performance than existing model-specific estimation methods.