Reconsidering Dependency Networks from an Information Geometry Perspective

TL;DR

This paper offers a geometric interpretation of dependency networks, providing theoretical bounds for their stationary distributions and demonstrating comparable accuracy to Bayesian networks with faster learning times.

Contribution

It introduces an information geometry perspective to dependency networks, enabling analysis of their stationary distributions and linking structure learning to optimization.

Findings

Dependency networks have similar distribution accuracy to Bayesian networks.

Dependency networks learn faster than Bayesian networks.

Theoretical bounds for stationary distributions are established.

Abstract

Dependency networks (Heckerman et al., 2000) are potential probabilistic graphical models for systems comprising a large number of variables. Like Bayesian networks, the structure of a dependency network is represented by a directed graph, and each node has a conditional probability table. Learning and inference are realized locally on individual nodes; therefore, computation remains tractable even with a large number of variables. However, the dependency network's learned distribution is the stationary distribution of a Markov chain called pseudo-Gibbs sampling and has no closed-form expressions. This technical disadvantage has impeded the development of dependency networks. In this paper, we consider a certain manifold for each node. Then, we can interpret pseudo-Gibbs sampling as iterative m-projections onto these manifolds. This interpretation provides a theoretical bound for the location where the stationary distribution of pseudo-Gibbs sampling exists in distribution space. Furthermore, this interpretation involves structure and parameter learning algorithms as optimization problems. In addition, we compare dependency and Bayesian networks experimentally. The results demonstrate that the dependency network and the Bayesian network have roughly the same performance in terms of the accuracy of their learned distributions. The results also show that the dependency network can learn much faster than the Bayesian network.