Tight Bounds for Jensen's Gap with Applications to Variational Inference

TL;DR

This paper introduces new bounds for Jensen's gap applicable to a wide range of functions and distributions, with implications for variational inference and probabilistic model generalization.

Contribution

We develop general, tight bounds for Jensen's gap that extend to exponential and logarithmic functions, enhancing understanding in variational inference.

Findings

Bounds are tighter than previous estimates.

Empirical validation confirms effectiveness.

Connections to PAC-Bayes improve understanding of generalization.

Abstract

Since its original formulation, Jensen's inequality has played a fundamental role across mathematics, statistics, and machine learning, with its probabilistic version highlighting the nonnegativity of the so-called Jensen's gap, i.e., the difference between the expectation of a convex function and the function at the expectation. Of particular importance is the case when the function is logarithmic, as this setting underpins many applications in variational inference, where the term variational gap is often used interchangeably. Recent research has focused on estimating the size of Jensen's gap and establishing tight lower and upper bounds under various assumptions on the underlying function and distribution, driven by practical challenges such as the intractability of log-likelihood in graphical models like variational autoencoders (VAEs). In this paper, we propose new, general bounds for Jensen's gap that accommodate a broad range of assumptions on both the function and the random variable, with special attention to exponential and logarithmic cases. We provide both analytical and empirical evidence for the performance of our method. Furthermore, we relate our bounds to the PAC-Bayes framework, providing new insights into generalization performance in probabilistic models.