Inversion of Bayesian Networks

TL;DR

This paper investigates the conditions under which recognition networks can exactly model true posterior distributions in Bayesian networks, providing both global and local criteria based on probabilistic graphical modeling principles.

Contribution

It establishes necessary and sufficient conditions for recognition networks to perfectly approximate posteriors in Bayesian networks, including new insights into local properties like perfectness.

Findings

Global conditions based on d-separation are identified.

Local conditions involve the property of perfectness at nodes.

Results clarify when recognition networks can exactly model true posteriors.

Abstract

Variational autoencoders and Helmholtz machines use a recognition network (encoder) to approximate the posterior distribution of a generative model (decoder). In this paper we study the necessary and sufficient properties of a recognition network so that it can model the true posterior distribution exactly. These results are derived in the general context of probabilistic graphical modelling / Bayesian networks, for which the network represents a set of conditional independence statements. We derive both global conditions, in terms of d-separation, and local conditions for the recognition network to have the desired qualities. It turns out that for the local conditions the property perfectness (for every node, all parents are joined) plays an important role.