Robustness Metric for Quantifying Causal Model Confidence and Parameter Uncertainty

TL;DR

This paper introduces a robustness metric for causal models that assesses confidence and parameter uncertainty, applicable to time series and non-time series data, validated on synthetic and real case data.

Contribution

A novel robustness metric for causal models that quantifies confidence and uncertainty, independent of the fitting method used.

Findings

Successfully identifies true data generating models with synthetic data.

Provides qualitative confidence levels and accurate coefficient uncertainty estimates.

Demonstrated effectiveness on simulated and real case study data.

Abstract

Many methods of estimating causal models do not provide estimates of confidence in the resulting model. In this work, a metric is proposed for validating the output of a causal model fit; the robustness of the model structure with resampled data. The metric is developed for time series causal models, but is also applicable to non-time series data. The proposed metric may be utilized regardless of the method selected for fitting the causal model. We find that with synthetically generated data, this metric is able to successfully identify the true data generating model in most cases. Additionally, the metric provides both a qualitative measure of model confidence represented by the robustness level as well as accurate estimates of uncertainty in model coefficients which are important in interpreting model results. The use of this metric is demonstrated on both numerically simulated data and a case study from existing causal model literature.