Hierarchical Quickest Change Detection via Surrogates

TL;DR

This paper introduces Hierarchical Quickest Change Detection (HQCD), a novel framework that leverages correlated surrogate sources to improve early detection of changepoints in time series data, enhancing robustness and understanding of underlying processes.

Contribution

The paper proposes a new hierarchical change detection framework that systematically incorporates surrogate sources, extending quickest detection theory to a hierarchical setting for better early detection.

Findings

HQCD improves robustness in changepoint detection.

HQCD effectively utilizes surrogate data like social media to predict target events.

Experimental results outperform state-of-the-art methods on synthetic and real data.

Abstract

Change detection (CD) in time series data is a critical problem as it reveal changes in the underlying generative processes driving the time series. Despite having received significant attention, one important unexplored aspect is how to efficiently utilize additional correlated information to improve the detection and the understanding of changepoints. We propose hierarchical quickest change detection (HQCD), a framework that formalizes the process of incorporating additional correlated sources for early changepoint detection. The core ideas behind HQCD are rooted in the theory of quickest detection and HQCD can be regarded as its novel generalization to a hierarchical setting. The sources are classified into targets and surrogates, and HQCD leverages this structure to systematically assimilate observed data to update changepoint statistics across layers. The decision on actual changepoints are provided by minimizing the delay while still maintaining reliability bounds. In addition, HQCD also uncovers interesting relations between changes at targets from changes across surrogates. We validate HQCD for reliability and performance against several state-of-the-art methods for both synthetic dataset (known changepoints) and several real-life examples (unknown changepoints). Our experiments indicate that we gain significant robustness without loss of detection delay through HQCD. Our real-life experiments also showcase the usefulness of the hierarchical setting by connecting the surrogate sources (such as Twitter chatter) to target sources (such as Employment related protests that ultimately lead to major uprisings).