Variational Inference with Locally Enhanced Bounds for Hierarchical Models

TL;DR

This paper introduces a novel variational inference approach with locally enhanced bounds for hierarchical models, enabling better accuracy and scalability through subsampling and independent tightening methods.

Contribution

It proposes a new family of variational bounds tailored for hierarchical models, allowing subsampling and independent application of tightening methods for improved inference.

Findings

Achieves more accurate posterior approximations than baseline methods.

Enables unbiased gradient estimation via subsampling.

Demonstrates improved scalability for large hierarchical models.

Abstract

Hierarchical models represent a challenging setting for inference algorithms. MCMC methods struggle to scale to large models with many local variables and observations, and variational inference (VI) may fail to provide accurate approximations due to the use of simple variational families. Some variational methods (e.g. importance weighted VI) integrate Monte Carlo methods to give better accuracy, but these tend to be unsuitable for hierarchical models, as they do not allow for subsampling and their performance tends to degrade for high dimensional models. We propose a new family of variational bounds for hierarchical models, based on the application of tightening methods (e.g. importance weighting) separately for each group of local random variables. We show that our approach naturally allows the use of subsampling to get unbiased gradients, and that it fully leverages the power of methods that build tighter lower bounds by applying them independently in lower dimensional spaces, leading to better results and more accurate posterior approximations than relevant baselines.