Affine calculus for constrained minima of the Kullback-Leibler divergence

TL;DR

This paper develops an affine calculus framework within information geometry to efficiently compute constrained minima of the Kullback-Leibler divergence, aiding various statistical machine learning applications.

Contribution

It introduces a non-parametric affine calculus based on statistical bundles for constrained KL divergence minimization in information geometry.

Findings

Provides a principled method for KL divergence minimization

Simplifies computations in mean-field approximation

Enhances understanding of variational Bayes and generative models

Abstract

The non-parametric version of Amari's dually affine Information Geometry provides a practical calculus to perform computations of interest in statistical machine learning. The method uses the notion of a statistical bundle, a mathematical structure that includes both probability densities and random variables to capture the spirit of Fisherian statistics. We focus on computations involving a constrained minimization of the Kullback-Leibler divergence. We show how to obtain neat and principled versions of known computation in applications such as mean-field approximation, adversarial generative models, and variational Bayes.