Sequential anomaly detection with sampling constraints

TL;DR

This paper introduces a new sequential anomaly detection framework with sampling constraints, allowing for unknown numbers of anomalies and flexible sampling, and proves asymptotic optimality of simple, computationally efficient policies.

Contribution

It formulates a novel anomaly detection problem with flexible bounds and sampling constraints, and establishes the asymptotic optimality of simple, adaptive policies.

Findings

Proposed a new formulation for sequential anomaly detection with flexible bounds.

Proved asymptotic optimality of a simple, adaptive sampling policy.

Demonstrated the policy's minimal computational complexity.

Abstract

The problem of sequential anomaly detection is considered, where multiple data sources are monitored in real time and the goal is to identify the "anomalous" ones among them, when it is not possible to sample all sources at all times. A detection scheme in this context requires specifying not only when to stop sampling and which sources to identify as anomalous upon stopping, but also which sources to sample at each time instance until stopping. A novel formulation for this problem is proposed, in which the number of anomalous sources is not necessarily known in advance and the number of sampled sources per time instance is not necessarily fixed. Instead, an arbitrary lower bound and an arbitrary upper bound are assumed on the number of anomalous sources, and the fraction of the expected number of samples over the expected time until stopping is required to not exceed an arbitrary, user-specified level. In addition to this sampling constraint, the probabilities of at least one false alarm and at least one missed detection are controlled below user-specified tolerance levels. A general criterion is established for a policy to achieve the minimum expected time until stopping to a first-order asymptotic approximation as both familywise error rates go to zero. This criterion is used to prove the asymptotic optimality of a family of policies that sample each source at each time instance with a probability that depends on the past observations only through the current estimate of the subset of anomalous sources. In particular, the asymptotic optimality is established of a policy that requires minimal computation under any setup of the problem.