Improve black-box sequential anomaly detector relevancy with limited user feedback

TL;DR

This paper presents a method to enhance the relevancy of black-box sequential anomaly detectors by leveraging limited user feedback to identify anomaly types and improve detection precision and recall.

Contribution

The method is detector-agnostic, uses minimal user feedback to classify anomaly types, and adjusts detection to improve relevancy and performance.

Findings

Significant improvements in precision and recall on synthetic datasets.

Effective identification of anomaly types from limited feedback.

Applicable across various black-box anomaly detectors.

Abstract

Anomaly detectors are often designed to catch statistical anomalies. End-users typically do not have interest in all of the detected outliers, but only those relevant to their application. Given an existing black-box sequential anomaly detector, this paper proposes a method to improve its user relevancy using a small number of human feedback. As our first contribution, the method is agnostic to the detector: it only assumes access to its anomaly scores, without requirement on any additional information inside it. Inspired by a fact that anomalies are of different types, our approach identifies these types and utilizes user feedback to assign relevancy to types. This relevancy score, as our second contribution, is used to adjust the subsequent anomaly selection process. Empirical results on synthetic and real-world datasets show that our approach yields significant improvements on precision and recall over a range of anomaly detectors.