How do some Bayesian Network machine learned graphs compare to causal knowledge?

TL;DR

This study compares purely machine learned Bayesian Network graphs with knowledge-based ones, revealing that knowledge-based graphs are more accurate predictors in limited data scenarios, despite machine learning algorithms achieving higher model scores.

Contribution

The paper provides a systematic comparison between machine learned and knowledge-based Bayesian Networks, highlighting the importance of causal knowledge in limited data contexts and evaluating the effectiveness of various learning algorithms.

Findings

Knowledge-based graphs are more accurate predictors with limited data.

Machine learning algorithms achieve higher model scores but deviate from true graphs.

Simulated data results do not reliably predict real-world performance.

Abstract

The graph of a Bayesian Network (BN) can be machine learned, determined by causal knowledge, or a combination of both. In disciplines like bioinformatics, applying BN structure learning algorithms can reveal new insights that would otherwise remain unknown. However, these algorithms are less effective when the input data are limited in terms of sample size, which is often the case when working with real data. This paper focuses on purely machine learned and purely knowledge-based BNs and investigates their differences in terms of graphical structure and how well the implied statistical models explain the data. The tests are based on four previous case studies whose BN structure was determined by domain knowledge. Using various metrics, we compare the knowledge-based graphs to the machine learned graphs generated from various algorithms implemented in TETRAD spanning all three classes of learning. The results show that, while the algorithms produce graphs with much higher model selection score, the knowledge-based graphs are more accurate predictors of variables of interest. Maximising score fitting is ineffective in the presence of limited sample size because the fitting becomes increasingly distorted with limited data, guiding algorithms towards graphical patterns that share higher fitting scores and yet deviate considerably from the true graph. This highlights the value of causal knowledge in these cases, as well as the need for more appropriate fitting scores suitable for limited data. Lastly, the experiments also provide new evidence that support the notion that results from simulated data tell us little about actual real-world performance.