Simulations evaluating resampling methods for causal discovery: ensemble performance and calibration

TL;DR

This paper evaluates resampling methods like subsampling and sampling with replacement for assessing confidence in causal discovery results through simulation studies, highlighting their effectiveness and calibration differences.

Contribution

It provides a systematic comparison of resampling techniques for confidence estimation in causal discovery, which is underexplored in practical applications.

Findings

Subsampling and sampling with replacement perform well as confidence indicators.

Calibration properties of resampling methods differ with sample size.

Choice of resampling method should depend on sample size.

Abstract

Causal discovery can be a powerful tool for investigating causality when a system can be observed but is inaccessible to experiments in practice. Despite this, it is rarely used in any scientific or medical fields. One of the major hurdles preventing the field of causal discovery from having a larger impact is that it is difficult to determine when the output of a causal discovery method can be trusted in a real-world setting. Trust is especially critical when human health is on the line. In this paper, we report the results of a series of simulation studies investigating the performance of different resampling methods as indicators of confidence in discovered graph features. We found that subsampling and sampling with replacement both performed surprisingly well, suggesting that they can serve as grounds for confidence in graph features. We also found that the calibration of subsampling and sampling with replacement had different convergence properties, suggesting that one's choice of which to use should depend on the sample size.