A Kernel Approach for Semi-implicit Variational Inference

TL;DR

This paper introduces KSIVI, a kernel-based semi-implicit variational inference method that simplifies optimization, improves expressiveness, and demonstrates strong empirical performance in Bayesian inference tasks.

Contribution

We propose KSIVI, a tractable kernel method for semi-implicit variational inference that avoids lower-level optimization and enhances expressiveness with hierarchical extensions.

Findings

KSIVI reduces optimization complexity via explicit solutions in RKHS.

The method achieves effective inference on synthetic and real-world tasks.

Theoretical guarantees include variance bounds and generalization bounds.

Abstract

Semi-implicit variational inference (SIVI) enhances the expressiveness of variational families through hierarchical semi-implicit distributions, but the intractability of their densities makes standard ELBO-based optimization biased. Recent score-matching approaches to SIVI (SIVI-SM) address this issue via a minimax formulation, at the expense of an additional lower-level optimization problem. In this paper, we propose kernel semi-implicit variational inference (KSIVI), a principled and tractable alternative that eliminates the lower-level optimization by leveraging kernel methods. We show that when optimizing over a reproducing kernel Hilbert space, the lower-level problem admits an explicit solution, reducing the objective to the kernel Stein discrepancy (KSD). Exploiting the hierarchical structure of semi-implicit distributions, the resulting KSD objective can be efficiently optimized using stochastic gradient methods. We establish optimization guarantees via variance bounds on Monte Carlo gradient estimators and derive statistical generalization bounds of order $\tilde{\mathcal{O}}(1/\sqrt{n})$. We further introduce a multi-layer hierarchical extension that improves expressiveness while preserving tractability. Empirical results on synthetic and real-world Bayesian inference tasks demonstrate the effectiveness of KSIVI.