Variational Cumulant Expansions for Intractable Distributions

TL;DR

This paper introduces a cumulant expansion method for approximating intractable distributions in probabilistic inference, improving accuracy and stability over traditional variational approaches, especially in Boltzmann machines.

Contribution

It presents a perturbational cumulant expansion technique that extends variational methods with higher-order corrections at low computational cost.

Findings

Enhanced accuracy in approximating Boltzmann machine distributions

Improved stability during learning processes

Demonstrated effectiveness of the method through simulations

Abstract

Intractable distributions present a common difficulty in inference within the probabilistic knowledge representation framework and variational methods have recently been popular in providing an approximate solution. In this article, we describe a perturbational approach in the form of a cumulant expansion which, to lowest order, recovers the standard Kullback-Leibler variational bound. Higher-order terms describe corrections on the variational approach without incurring much further computational cost. The relationship to other perturbational approaches such as TAP is also elucidated. We demonstrate the method on a particular class of undirected graphical models, Boltzmann machines, for which our simulation results confirm improved accuracy and enhanced stability during learning.