Estimating Well-Performing Bayesian Networks using Bernoulli Mixtures

TL;DR

This paper introduces Bernoulli mixture networks (BMNs), a new approach for estimating Bayesian network parameters that enhances model flexibility and accuracy by representing CPDs as mixtures of local distributions, leading to improved test performance.

Contribution

The paper proposes Bernoulli mixture networks for Bayesian networks, allowing finer-grained dependencies and better data fit compared to traditional models.

Findings

BMNs reduce overfitting by limiting mixture complexity.

Simple substructures in BMNs perform nearly as well as complex ones.

BMNs significantly outperform conventional Bayesian networks in real-world data.

Abstract

A novel method for estimating Bayesian network (BN) parameters from data is presented which provides improved performance on test data. Previous research has shown the value of representing conditional probability distributions (CPDs) via neural networks(Neal 1992), noisy-OR gates (Neal 1992, Diez 1993)and decision trees (Friedman and Goldszmidt 1996).The Bernoulli mixture network (BMN) explicitly represents the CPDs of discrete BN nodes as mixtures of local distributions,each having a different set of parents.This increases the space of possible structures which can be considered,enabling the CPDs to have finer-grained dependencies.The resulting estimation procedure induces a modelthat is better able to emulate the underlying interactions occurring in the data than conventional conditional Bernoulli network models.The results for artificially generated data indicate that overfitting is best reduced by restricting the complexity of candidate mixture substructures local to each node. Furthermore, mixtures of very simple substructures can perform almost as well as more complex ones.The BMN is also applied to data collected from an online adventure game with an application to keyhole plan recognition. The results show that the BMN-based model brings a dramatic improvement in performance over a conventional BN model.