Quantitative probing: Validating causal models using quantitative domain knowledge

TL;DR

Quantitative probing offers a model-agnostic framework for validating causal models by integrating quantitative domain knowledge, enhancing current validation strategies through simulation and analysis of failure scenarios.

Contribution

The paper introduces a novel, model-agnostic validation method called quantitative probing that incorporates quantitative domain knowledge into causal model validation.

Findings

Effective validation demonstrated on Pearl's sprinkler example

Simulation studies reveal strengths and limitations of the method

Open-source Python packages provided for implementation

Abstract

We present quantitative probing as a model-agnostic framework for validating causal models in the presence of quantitative domain knowledge. The method is constructed as an analogue of the train/test split in correlation-based machine learning and as an enhancement of current causal validation strategies that are consistent with the logic of scientific discovery. The effectiveness of the method is illustrated using Pearl's sprinkler example, before a thorough simulation-based investigation is conducted. Limits of the technique are identified by studying exemplary failing scenarios, which are furthermore used to propose a list of topics for future research and improvements of the presented version of quantitative probing. The code for integrating quantitative probing into causal analysis, as well as the code for the presented simulation-based studies of the effectiveness of quantitative probing is provided in two separate open-source Python packages.