Causal datasheet: An approximate guide to practically assess Bayesian networks in the real world

TL;DR

This paper introduces a Causal Datasheet framework that estimates Bayesian Network performance on datasets, aiding validation and interpretation in real-world causal analysis, demonstrated through synthetic data and a maternal health survey.

Contribution

It proposes a novel Causal Datasheet concept with a prototype tool to assess Bayesian Network reliability on specific datasets, enhancing practical causal inference validation.

Findings

Generated over 30,000 synthetic datasets for benchmarking

Automatically populated datasheets with performance expectations

Applied to maternal health survey in Uttar Pradesh

Abstract

In solving real-world problems like changing healthcare-seeking behaviors, designing interventions to improve downstream outcomes requires an understanding of the causal links within the system. Causal Bayesian Networks (BN) have been proposed as one such powerful method. In real-world applications, however, confidence in the results of BNs are often moderate at best. This is due in part to the inability to validate against some ground truth, as the DAG is not available. This is especially problematic if the learned DAG conflicts with pre-existing domain doctrine. At the policy level, one must justify insights generated by such analysis, preferably accompanying them with uncertainty estimation. Here we propose a causal extension to the datasheet concept proposed by Gebru et al (2018) to include approximate BN performance expectations for any given dataset. To generate the results for a prototype Causal Datasheet, we constructed over 30,000 synthetic datasets with properties mirroring characteristics of real data. We then recorded the results given by state-of-the-art structure learning algorithms. These results were used to populate the Causal Datasheet, and recommendations were automatically generated dependent on expected performance. As a proof of concept, we used our Causal Datasheet Generation Tool (CDG-T) to assign expected performance expectations to a maternal health survey we conducted in Uttar Pradesh, India.