Kernel Semi-Implicit Variational Inference

TL;DR

This paper introduces kernel SIVI (KSIVI), a novel variational inference method that leverages kernel tricks to eliminate lower-level optimization, enabling efficient and unbiased training for semi-implicit variational distributions.

Contribution

KSIVI provides an explicit solution for the lower-level problem in semi-implicit variational inference using kernel methods, simplifying training and improving convergence guarantees.

Findings

KSIVI achieves efficient Bayesian inference on synthetic and real datasets.

It provides convergence guarantees with a variance bound for gradient estimators.

KSIVI outperforms previous semi-implicit variational methods in accuracy and speed.

Abstract

Semi-implicit variational inference (SIVI) extends traditional variational families with semi-implicit distributions defined in a hierarchical manner. Due to the intractable densities of semi-implicit distributions, classical SIVI often resorts to surrogates of evidence lower bound (ELBO) that would introduce biases for training. A recent advancement in SIVI, named SIVI-SM, utilizes an alternative score matching objective made tractable via a minimax formulation, albeit requiring an additional lower-level optimization. In this paper, we propose kernel SIVI (KSIVI), a variant of SIVI-SM that eliminates the need for lower-level optimization through kernel tricks. Specifically, we show that when optimizing over a reproducing kernel Hilbert space (RKHS), the lower-level problem has an explicit solution. This way, the upper-level objective becomes the kernel Stein discrepancy (KSD), which is readily computable for stochastic gradient descent due to the hierarchical structure of semi-implicit variational distributions. An upper bound for the variance of the Monte Carlo gradient estimators of the KSD objective is derived, which allows us to establish novel convergence guarantees of KSIVI. We demonstrate the effectiveness and efficiency of KSIVI on both synthetic distributions and a variety of real data Bayesian inference tasks.