Large-Scale Multi-Stream Quickest Change Detection via Shrinkage Post-Change Estimation

TL;DR

This paper introduces a scalable method for quickest change detection in large-scale data streams, utilizing shrinkage estimators to improve detection delay performance in sparse and dense change scenarios.

Contribution

It develops a systematic approach combining thresholding and shrinkage estimators for efficient global change detection in large-scale data streams.

Findings

Shrinkage estimation balances detection delay tradeoffs.

Method effective in sparse and dense change scenarios.

Numerical simulations confirm theoretical advantages.

Abstract

The quickest change detection problem is considered in the context of monitoring large-scale independent normal distributed data streams with possible changes in some of the means. It is assumed that for each individual local data stream, either there are no local changes, or there is a "big" local change that is larger than a pre-specified lower bound. Two different kinds of scenarios are studied: one is the sparse post-change case when the unknown number of affected data streams is much smaller than the total number of data streams, and the other is when all local data streams are affected simultaneously although not necessarily identically. We propose a systematic approach to develop efficient global monitoring schemes for quickest change detection by combining hard thresholding with linear shrinkage estimators to estimating all post-change parameters simultaneously. Our theoretical analysis demonstrates that the shrinkage estimation can balance the tradeoff between the first-order and second-order terms of the asymptotic expression on the detection delays, and our numerical simulation studies illustrate the usefulness of shrinkage estimation and the challenge of Monte Carlo simulation of the average run length to false alarm in the context of online monitoring large-scale data streams.