Scalable Linear Causal Inference for Irregularly Sampled Time Series with Long Range Dependencies

TL;DR

This paper introduces a frequency domain framework for scalable linear causal inference that effectively handles irregular sampling, long-range dependencies, and distributed processing, demonstrated on simulations and financial data.

Contribution

It presents a novel frequency domain approach for causal analysis that overcomes key challenges of irregular sampling and long-range dependencies in large-scale time series.

Findings

Accurately recovers causal structure in simulations.

Effective in high-frequency financial data.

Handles irregular sampling and long-range dependencies.

Abstract

Linear causal analysis is central to a wide range of important application spanning finance, the physical sciences, and engineering. Much of the existing literature in linear causal analysis operates in the time domain. Unfortunately, the direct application of time domain linear causal analysis to many real-world time series presents three critical challenges: irregular temporal sampling, long range dependencies, and scale. Moreover, real-world data is often collected at irregular time intervals across vast arrays of decentralized sensors and with long range dependencies which make naive time domain correlation estimators spurious. In this paper we present a frequency domain based estimation framework which naturally handles irregularly sampled data and long range dependencies while enabled memory and communication efficient distributed processing of time series data. By operating in the frequency domain we eliminate the need to interpolate and help mitigate the effects of long range dependencies. We implement and evaluate our new work-flow in the distributed setting using Apache Spark and demonstrate on both Monte Carlo simulations and high-frequency financial trading that we can accurately recover causal structure at scale.