Towards Dynamic Causal Discovery with Rare Events: A Nonparametric Conditional Independence Test

TL;DR

This paper introduces a new nonparametric conditional independence test tailored for dynamic causal discovery in systems with rare events, addressing limitations of existing methods in uncovering causality during low-probability occurrences.

Contribution

The paper presents a novel statistical independence test leveraging time-invariance to detect causal links associated with rare events, with theoretical guarantees and empirical validation.

Findings

Effective in identifying causal links during rare events

Provides non-asymptotic sample complexity bounds

Validated on simulated and real-world datasets

Abstract

Causal phenomena associated with rare events occur across a wide range of engineering problems, such as risk-sensitive safety analysis, accident analysis and prevention, and extreme value theory. However, current methods for causal discovery are often unable to uncover causal links, between random variables in a dynamic setting, that manifest only when the variables first experience low-probability realizations. To address this issue, we introduce a novel statistical independence test on data collected from time-invariant dynamical systems in which rare but consequential events occur. In particular, we exploit the time-invariance of the underlying data to construct a superimposed dataset of the system state before rare events happen at different timesteps. We then design a conditional independence test on the reorganized data. We provide non-asymptotic sample complexity bounds for the consistency of our method, and validate its performance across various simulated and real-world datasets, including incident data collected from the Caltrans Performance Measurement System (PeMS). Code containing the datasets and experiments is publicly available.