A Fast and Efficient Change-point Detection Framework based on Approximate $k$-Nearest Neighbor Graphs

TL;DR

This paper introduces a fast, scalable change-point detection method using approximate k-nearest neighbor graphs, suitable for high-dimensional big data sequences, with proven control of false positives and demonstrated effectiveness on neuroimaging datasets.

Contribution

It proposes a novel change-point detection framework leveraging approximate k-NN graphs with an analytic error control formula, improving speed and adaptability for high-dimensional data.

Findings

Method has complexity $O(dn(\log n + k \log d) + nk^2)$.

Effectively detects various change types in high-dimensional data.

Demonstrated success on fMRI and Neuropixels datasets.

Abstract

Change-point analysis is thriving in this big data era to address problems arising in many fields where massive data sequences are collected to study complicated phenomena over time. It plays an important role in processing these data by segmenting a long sequence into homogeneous parts for follow-up studies. The task requires the method to be able to process large datasets quickly and deal with various types of changes for high-dimensional data. We propose a new approach making use of approximate $k$-nearest neighbor information from the observations, and derive an analytic formula to control the type I error. The time complexity of our proposed method is $O\left(dn(\log n+k \log d)+nk^2\right)$ for an $n$-length sequence of $d$-dimensional data. The test statistic we consider incorporates a useful pattern for moderate- to high- dimensional data so that the proposed method could detect various types of changes in the sequence. The new approach is also asymptotic distribution free, facilitating its usage for a broader community. We apply our method to fMRI datasets and Neuropixels datasets to illustrate its effectiveness.