Stochastic complexity of Bayesian networks

TL;DR

This paper analyzes the statistical properties of Bayesian networks with latent variables using algebraic geometry, revealing their stochastic complexity and implications for model selection and generalization error.

Contribution

It applies algebraic geometric methods to clarify the stochastic complexity of Bayesian networks, showing their generalization error is smaller than regular models and suggesting improvements to BIC.

Findings

Upper bound of stochastic complexity is smaller than parameter space dimension

Bayesian networks have smaller generalization error than regular models

BIC criterion needs modification for Bayesian networks

Abstract

Bayesian networks are now being used in enormous fields, for example, diagnosis of a system, data mining, clustering and so on. In spite of their wide range of applications, the statistical properties have not yet been clarified, because the models are nonidentifiable and non-regular. In a Bayesian network, the set of its parameter for a smaller model is an analytic set with singularities in the space of large ones. Because of these singularities, the Fisher information matrices are not positive definite. In other words, the mathematical foundation for learning was not constructed. In recent years, however, we have developed a method to analyze non-regular models using algebraic geometry. This method revealed the relation between the models singularities and its statistical properties. In this paper, applying this method to Bayesian networks with latent variables, we clarify the order of the stochastic complexities.Our result claims that the upper bound of those is smaller than the dimension of the parameter space. This means that the Bayesian generalization error is also far smaller than that of regular model, and that Schwarzs model selection criterion BIC needs to be improved for Bayesian networks.