Likelihood Scores for Sparse Signal and Change-Point Detection

TL;DR

This paper introduces a likelihood ratio score for detecting sparse signals and change-points in large data streams, optimizing detection power at the boundary of detectability and outperforming traditional thresholding methods.

Contribution

It proposes a new sparse likelihood score based on the Neyman-Pearson lemma, achieving second-order optimality in detecting minimal change magnitudes in sparse settings.

Findings

Score is successful at the boundary of detectable signals.

Achieves second-order optimality in detection.

Down-weights small change signals more effectively.

Abstract

We consider here the identification of change-points on large-scale data streams. The objective is to find the most efficient way of combining information across data stream so that detection is possible under the smallest detectable change magnitude. The challenge comes from the sparsity of change-points when only a small fraction of data streams undergo change at any point in time. The most successful approach to the sparsity issue so far has been the application of hard thresholding such that only local scores from data streams exhibiting significant changes are considered and added. However the identification of an optimal threshold is a difficult one. In particular it is unlikely that the same threshold is optimal for different levels of sparsity. We propose here a sparse likelihood score for identifying a sparse signal. The score is a likelihood ratio for testing between the null hypothesis of no change against an alternative hypothesis in which the change-points or signals are barely detectable. By the Neyman-Pearson Lemma this score has maximum detection power at the given alternative. The outcome is that we have a scoring of data streams that is successful in detecting at the boundary of the detectable region of signals and change-points. The likelihood score can be seen as a soft thresholding approach to sparse signal and change-point detection in which local scores that indicate small changes are down-weighted much more than local scores indicating large changes. We are able to show second-order optimality of the sparsity likelihood score in the sense of achieving successful detection at the minimum detectable order of change magnitude as well as at the minimum detection asymptotic constant with respect this order of change.