Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data (Revised)

TL;DR

This paper introduces INCAD, a streaming clustering and anomaly detection algorithm that operates without predefined thresholds or cluster counts, providing reliable anomaly detection by separating normal and abnormal behaviors.

Contribution

The paper presents a novel streaming clustering and anomaly detection method that does not rely on thresholds or known cluster numbers, improving robustness and reliability.

Findings

Operates without thresholds or prior cluster knowledge

Ensures cluster formation is unaffected by anomalies

Provides probabilistic anomaly detection in streaming data

Abstract

Most current clustering based anomaly detection methods use scoring schema and thresholds to classify anomalies. These methods are often tailored to target specific data sets with "known" number of clusters. The paper provides a streaming clustering and anomaly detection algorithm that does not require strict arbitrary thresholds on the anomaly scores or knowledge of the number of clusters while performing probabilistic anomaly detection and clustering simultaneously. This ensures that the cluster formation is not impacted by the presence of anomalous data, thereby leading to more reliable definition of "normal vs abnormal" behavior. The motivations behind developing the INCAD model and the path that leads to the streaming model is discussed.