Entropy Message Passing

TL;DR

The paper introduces entropy message passing (EMP), a novel algorithm for cycle-free factor graphs that efficiently computes entropy and related expressions, enhancing inference methods in probabilistic models.

Contribution

It presents EMP as a new message passing algorithm based on the entropy semiring, extending sum-product algorithms to compute entropy and expectations in probabilistic models.

Findings

EMP efficiently computes entropy in cycle-free factor graphs.

EMP can be applied to expectation maximization and gradient descent.

The algorithm bridges automata theory and probabilistic inference.

Abstract

The paper proposes a new message passing algorithm for cycle-free factor graphs. The proposed "entropy message passing" (EMP) algorithm may be viewed as sum-product message passing over the entropy semiring, which has previously appeared in automata theory. The primary use of EMP is to compute the entropy of a model. However, EMP can also be used to compute expressions that appear in expectation maximization and in gradient descent algorithms.