Sequential Change Detection in Correlation Structures with Window-Limited Statistics

TL;DR

This paper introduces a new method for detecting change points in the correlation structure of streaming data using window-limited statistics, with theoretical guarantees and applications to real-world datasets.

Contribution

It proposes novel detection statistics for dense and sparse changes, along with an efficient thresholding algorithm based on sign-flip permutations, applicable in high-dimensional settings.

Findings

Effective detection with small delays comparable to CUSUM

High accuracy in real-world applications like El Niño forecasting

Theoretical guarantees on detection performance

Abstract

We consider detecting change points in the correlation structure of streaming data with minimum assumptions posed on the underlying data distribution. Detection statistics are constructed for dense and sparse change settings, based on $\ell_1$ and $\ell_{\infty}$ norms of the squared difference of vectorized pre- and post-change correlation matrices, respectively. We also propose a novel threshold determination algorithm based on sign-flip permutations that enhances the efficiency of our procedure, particularly when the data dimension is large compared to the window size. Theoretical guarantees of the proposed methods are provided in terms of average run length in the no-change regime and expected detection delay in the post-change regime. We evaluate the performance of the proposed methods across a wide range of simulated datasets and demonstrate their effectiveness, with small detection delays that are comparable to the exact optimal CUSUM test. Finally, we demonstrate the effectiveness of our methods on real-world datasets, including El Ni{\~n}o event forecasting, where we achieve a state-of-the-art hit rate exceeding 0.86 with near-zero false alarms, as well as seismic event detection.