Anomaly Search over Composite Hypotheses in Hierarchical Statistical Models

TL;DR

This paper introduces Hierarchical Dynamic Search (HDS), a sequential anomaly detection method in hierarchical models that is order-optimal, asymptotically optimal, and effective in real-world cybersecurity datasets.

Contribution

The paper proposes HDS, a novel hierarchical search strategy using GLLR statistics, optimized for composite hypotheses and multiple anomalies detection.

Findings

HDS is order-optimal in search space

HDS is asymptotically optimal for detection accuracy

HDS outperforms existing methods on DARPA dataset

Abstract

Detection of anomalies among a large number of processes is a fundamental task that has been studied in multiple research areas, with diverse applications spanning from spectrum access to cyber-security. Anomalous events are characterized by deviations in data distributions, and thus can be inferred from noisy observations based on statistical methods. In some scenarios, one can often obtain noisy observations aggregated from a chosen subset of processes. Such hierarchical search can further minimize the sample complexity while retaining accuracy. An anomaly search strategy should thus be designed based on multiple requirements, such as maximizing the detection accuracy; efficiency, be efficient in terms of sample complexity; and be able to cope with statistical models that are known only up to some missing parameters (i.e., composite hypotheses). In this paper, we consider anomaly detection with observations taken from a chosen subset of processes that conforms to a predetermined tree structure with partially known statistical model. We propose Hierarchical Dynamic Search (HDS), a sequential search strategy that uses two variations of the Generalized Log Likelihood Ratio (GLLR) statistic, and can be used for detection of multiple anomalies. HDS is shown to be order-optimal in terms of the size of the search space, and asymptotically optimal in terms of detection accuracy. An explicit upper bound on the error probability is established for the finite sample regime. In addition to extensive experiments on synthetic datasets, experiments have been conducted on the DARPA intrusion detection dataset, showing that HDS is superior to existing methods.