Local Max-Entropy and Free Energy Principles, Belief Diffusions and their Singularities

TL;DR

This paper explores the relationships between Bethe-Kikuchi variational principles, belief propagation algorithms, and belief diffusions, analyzing their singularities and stability through polynomial equations and loop series expansions.

Contribution

It generalizes belief propagation to continuous-time diffusions based on variational principles and characterizes the singularities and stability of beliefs using polynomial equations.

Findings

Critical points lie at intersections of constraint surfaces.

Singular beliefs form a hypersurface described by polynomial equations.

Loop series expansion expresses the polynomial for binary graphs.

Abstract

A comprehensive picture of three Bethe-Kikuchi variational principles including their relationship to belief propagation (BP) algorithms on hypergraphs is given. The structure of BP equations is generalized to define continuous-time diffusions, solving localized versions of the max-entropy principle (A), the variational free energy principle (B), and a less usual equilibrium free energy principle (C), Legendre dual to A. Both critical points of Bethe-Kikuchi functionals and stationary beliefs are shown to lie at the non-linear intersection of two constraint surfaces, enforcing energy conservation and marginal consistency respectively. The hypersurface of singular beliefs, accross which equilibria become unstable as the constraint surfaces meet tangentially, is described by polynomial equations in the convex polytope of consistent beliefs. This polynomial is expressed by a loop series expansion for graphs of binary variables.