Semi-Implicit Variational Inference via Score Matching

TL;DR

This paper introduces SIVI-SM, a novel training method for semi-implicit variational inference that uses score matching, enabling more accurate and efficient Bayesian inference without expensive sampling.

Contribution

We propose SIVI-SM, a score matching-based training approach for semi-implicit variational inference that handles intractable densities more effectively than existing methods.

Findings

SIVI-SM closely matches MCMC accuracy.

SIVI-SM outperforms ELBO-based SIVI methods.

Efficient training without inner-loop MCMC.

Abstract

Semi-implicit variational inference (SIVI) greatly enriches the expressiveness of variational families by considering implicit variational distributions defined in a hierarchical manner. However, due to the intractable densities of variational distributions, current SIVI approaches often use surrogate evidence lower bounds (ELBOs) or employ expensive inner-loop MCMC runs for unbiased ELBOs for training. In this paper, we propose SIVI-SM, a new method for SIVI based on an alternative training objective via score matching. Leveraging the hierarchical structure of semi-implicit variational families, the score matching objective allows a minimax formulation where the intractable variational densities can be naturally handled with denoising score matching. We show that SIVI-SM closely matches the accuracy of MCMC and outperforms ELBO-based SIVI methods in a variety of Bayesian inference tasks.