Quick Search for Rare Events

TL;DR

This paper introduces a statistical framework for efficiently detecting rare events in large datasets by adaptively focusing on relevant segments, balancing detection reliability and agility.

Contribution

It proposes an adaptive sampling procedure tailored for Gaussian signals to improve rare event detection in large, noisy datasets.

Findings

Effective detection of rare Gaussian signals with unusual mean and variance.

Balanced approach optimizing detection reliability and speed.

Framework applicable to various applications involving rare event detection.

Abstract

Rare events can potentially occur in many applications. When manifested as opportunities to be exploited, risks to be ameliorated, or certain features to be extracted, such events become of paramount significance. Due to their sporadic nature, the information-bearing signals associated with rare events often lie in a large set of irrelevant signals and are not easily accessible. This paper provides a statistical framework for detecting such events so that an optimal balance between detection reliability and agility, as two opposing performance measures, is established. The core component of this framework is a sampling procedure that adaptively and quickly focuses the information-gathering resources on the segments of the dataset that bear the information pertinent to the rare events. Particular focus is placed on Gaussian signals with the aim of detecting signals with rare mean and variance values.