Causal-INSIGHT: Probing Temporal Models to Extract Causal Structure

TL;DR

Causal-INSIGHT is a versatile framework that interprets trained temporal models to reveal their implied causal influence structures by analyzing responses to intervention-inspired input clamping, improving interpretability and delay localization.

Contribution

It introduces a model-agnostic, post-hoc method for extracting directed, time-lagged influence structures from pre-trained temporal predictors, without needing ground-truth causal graphs.

Findings

Generalizes across diverse architectures

Maintains competitive structural accuracy

Improves temporal delay localization

Abstract

Understanding directed temporal interactions in multivariate time series is essential for interpreting complex dynamical systems and the predictive models trained on them. We present Causal-INSIGHT, a model-agnostic, post-hoc interpretation framework for extracting model-implied (predictor-dependent), directed, time-lagged influence structure from trained temporal predictors. Rather than inferring causal structure at the level of the data-generating process, Causal-INSIGHT analyzes how a fixed, pre-trained predictor responds to systematic, intervention-inspired input clamping applied at inference time. From these responses, we construct directed temporal influence signals that reflect the dependencies the predictor relies on for prediction, and introduce Qbic, a sparsity-aware graph selection criterion that balances predictive fidelity and structural complexity without requiring ground-truth graph labels. Experiments across synthetic, simulated, and realistic benchmarks show that Causal-INSIGHT generalizes across diverse backbone architectures, maintains competitive structural accuracy, and yields significant improvements in temporal delay localization when applied to existing predictors.