Sparse, self-organizing ensembles of local kernels detect rare statistical anomalies

TL;DR

This paper introduces SparKer, a sparse ensemble of local kernels designed for anomaly detection in high-dimensional data, emphasizing self-organization, efficiency, and interpretability to identify rare anomalies effectively.

Contribution

The paper proposes a novel self-organizing kernel ensemble method, SparKer, that adaptively models likelihood ratios for anomaly detection with theoretical insights and practical validation.

Findings

Ensembles with few kernels detect anomalies in thousands of dimensions.

SparKer effectively identifies statistically significant anomalies.

The approach is scalable and interpretable across scientific domains.

Abstract

Modern artificial intelligence has revolutionized our ability to extract rich and versatile data representations across scientific disciplines. Yet, the statistical properties of these representations remain poorly controlled, causing misspecified anomaly detection (AD) methods to falter. Weak or rare signals can remain hidden within the apparent regularity of normal data, creating a gap in our ability to detect and interpret anomalies. We examine this gap and identify a set of structural desiderata for detection methods operating under minimal prior information: sparsity, to enforce parsimony; locality, to preserve geometric sensitivity; and competition, to promote efficient allocation of model capacity. These principles define a class of self-organizing local kernels that adaptively partition the representation space around regions of statistical imbalance. As an instantiation of these principles, we introduce SparKer, a sparse ensemble of Gaussian kernels trained within a semi-supervised Neyman--Pearson framework to locally model the likelihood ratio between a sample that may contain anomalies and a nominal, anomaly-free reference. We provide theoretical insights into the mechanisms that drive detection and self-organization in the proposed model, and demonstrate the effectiveness of this approach on realistic high-dimensional problems of scientific discovery, open-world novelty detection, intrusion detection, and generative-model validation. Our applications span both the natural- and computer-science domains. We demonstrate that ensembles containing only a handful of kernels can identify statistically significant anomalous locations within representation spaces of thousands of dimensions, underscoring both the interpretability, efficiency and scalability of the proposed approach.