Statistical Guarantees for Transformation Based Models with Applications to Implicit Variational Inference

TL;DR

This paper provides theoretical guarantees for transformation-based models, especially in non-parametric inference and implicit variational inference, demonstrating their support, concentration rates, and approximation capabilities.

Contribution

It offers the first theoretical justification for implicit variational inference using transformation-based models with Gaussian process priors.

Findings

Support of transformation prior is large in L1 sense

Posterior concentrates at near-optimal rate with GP prior

GP-IVI achieves optimal risk bounds and KL divergence approximation

Abstract

Transformation-based methods have been an attractive approach in non-parametric inference for problems such as unconditional and conditional density estimation due to their unique hierarchical structure that models the data as flexible transformation of a set of common latent variables. More recently, transformation-based models have been used in variational inference (VI) to construct flexible implicit families of variational distributions. However, their use in both non-parametric inference and variational inference lacks theoretical justification. We provide theoretical justification for the use of non-linear latent variable models (NL-LVMs) in non-parametric inference by showing that the support of the transformation induced prior in the space of densities is sufficiently large in the $L_1$ sense. We also show that, when a Gaussian process (GP) prior is placed on the transformation function, the posterior concentrates at the optimal rate up to a logarithmic factor. Adopting the flexibility demonstrated in the non-parametric setting, we use the NL-LVM to construct an implicit family of variational distributions, deemed GP-IVI. We delineate sufficient conditions under which GP-IVI achieves optimal risk bounds and approximates the true posterior in the sense of the Kullback-Leibler divergence. To the best of our knowledge, this is the first work on providing theoretical guarantees for implicit variational inference.