Sequential Multivariate Change Detection with Calibrated and Memoryless False Detection Rates

TL;DR

This paper introduces a simulation-based method for setting time-varying thresholds in sequential change detection, ensuring calibrated false positive rates and targeted expected runtimes, improving practical reliability over existing asymptotic approaches.

Contribution

It proposes a novel simulation-based approach for dynamic threshold setting that maintains constant false positive rates and accurate expected runtimes in sequential change detection.

Findings

Achieves accurate false positive rate calibration across time.

Reduces computational complexity of MMD estimator from quadratic to linear.

Provides a practical framework for reliable sequential change detection.

Abstract

Responding appropriately to the detections of a sequential change detector requires knowledge of the rate at which false positives occur in the absence of change. Setting detection thresholds to achieve a desired false positive rate is challenging. Existing works resort to setting time-invariant thresholds that focus on the expected runtime of the detector in the absence of change, either bounding it loosely from below or targeting it directly but with asymptotic arguments that we show cause significant miscalibration in practice. We present a simulation-based approach to setting time-varying thresholds that allows a desired expected runtime to be accurately targeted whilst additionally keeping the false positive rate constant across time steps. Whilst the approach to threshold setting is metric agnostic, we show how the cost of using the popular quadratic time MMD estimator can be reduced from $O(N^2B)$ to $O(N^2+NB)$ during configuration and from $O(N^2)$ to $O(N)$ during operation, where $N$ and $B$ are the numbers of reference and bootstrap samples respectively.