Note on the bias and variance of variational inference

TL;DR

This paper explores the relationship between the bias in variational inference and the variance of the likelihood ratio, proposing that reducing variance can decrease bias.

Contribution

It establishes an upper bound on the variational gap based on the dispersion of the likelihood ratio, offering insights for bias reduction techniques.

Findings

Bias can be reduced by making the likelihood ratio distribution more concentrated.

The variational gap is upper bounded by a dispersion measure of the likelihood ratio.

Variance reduction methods can potentially improve variational inference accuracy.

Abstract

In this note, we study the relationship between the variational gap and the variance of the (log) likelihood ratio. We show that the gap can be upper bounded by some form of dispersion measure of the likelihood ratio, which suggests the bias of variational inference can be reduced by making the distribution of the likelihood ratio more concentrated, such as via averaging and variance reduction.