Hierarchical Reference Sets for Robust Unsupervised Detection of Scattered and Clustered Outliers

TL;DR

This paper introduces a graph-based outlier detection method for IoT data that effectively distinguishes between scattered and clustered anomalies, improving robustness in unsupervised analysis tasks.

Contribution

It proposes a novel outlier detection paradigm using graph structures to differentiate scattered and clustered outliers in IoT data, enhancing detection accuracy.

Findings

Effective detection of scattered outliers without interference from clusters

Ability to identify and isolate clustered outlier groups

Validated through extensive experiments and downstream tasks

Abstract

Most real-world IoT data analysis tasks, such as clustering and anomaly event detection, are unsupervised and highly susceptible to the presence of outliers. In addition to sporadic scattered outliers caused by factors such as faulty sensor readings, IoT systems often exhibit clustered outliers. These occur when multiple devices or nodes produce similar anomalous measurements, for instance, owing to localized interference, emerging security threats, or regional false alarms, forming micro-clusters. These clustered outliers can be easily mistaken for normal behavior because of their relatively high local density, thereby obscuring the detection of both scattered and contextual anomalies. To address this, we propose a novel outlier detection paradigm that leverages the natural neighboring relationships using graph structures. This facilitates multi-perspective anomaly evaluation by incorporating reference sets at both local and global scales derived from the graph. Our approach enables the effective recognition of scattered outliers without interference from clustered anomalies, whereas the graph structure simultaneously helps reflect and isolate clustered outlier groups. Extensive experiments, including comparative performance analysis, ablation studies, validation on downstream clustering tasks, and evaluation of hyperparameter sensitivity, demonstrate the efficacy of the proposed method. The source code is available at https://github.com/gordonlok/DROD.