CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios

TL;DR

CausalCompass is a benchmark framework for evaluating the robustness of time-series causal discovery methods under assumption violations, revealing deep learning approaches often perform best across diverse scenarios.

Contribution

We introduce CausalCompass, a flexible benchmark for systematically assessing TSCD methods' robustness under assumption violations, including extensive experiments and analysis.

Findings

Deep learning-based methods outperform others across various assumption-violation scenarios.

NTS-NOTEARS performs poorly without preprocessing but excels after standardization.

No single method is best in all tested scenarios.

Abstract

Causal discovery from time series is a fundamental task in machine learning. However, its widespread adoption is hindered by a reliance on untestable causal assumptions and by the lack of robustness-oriented evaluation in existing benchmarks. To address these challenges, we propose CausalCompass, a flexible and extensible benchmark framework designed to assess the robustness of time-series causal discovery (TSCD) methods under violations of modeling assumptions. To demonstrate the practical utility of CausalCompass, we conduct extensive benchmarking of representative TSCD algorithms across eight assumption-violation scenarios. Our experimental results indicate that no single method consistently attains optimal performance across all settings. Nevertheless, the methods exhibiting superior overall performance across diverse scenarios are almost invariably deep learning-based approaches. We further provide hyperparameter sensitivity analyses to deepen the understanding of these findings. We additionally conduct ablation experiments to explain the strong performance of deep learning-based methods under assumption violations. We also find, somewhat surprisingly, that NTS-NOTEARS relies heavily on standardized preprocessing in practice, performing poorly in the vanilla setting but exhibiting strong performance after standardization. Finally, our work aims to provide a comprehensive and systematic evaluation of TSCD methods under assumption violations, thereby facilitating their broader adoption in real-world applications. The user-friendly implementation, documentation and datasets are available at https://anonymous.4open.science/r/CausalCompass-anonymous-5B4F/.