SDOoop: Capturing Periodical Patterns and Out-of-phase Anomalies in Streaming Data Analysis

TL;DR

SDOoop is an advanced streaming data analysis method that detects complex temporal and contextual anomalies, providing interpretable insights with high efficiency suitable for big data applications.

Contribution

It extends the SDO anomaly detection method to retain temporal information, enabling detection of out-of-phase anomalies and analysis of data geometries and patterns.

Findings

SDOoop detects contextual anomalies undetectable by traditional methods.

It performs comparably or better than state-of-the-art approaches in various domains.

Operates with constant space and time complexity, suitable for large-scale streaming data.

Abstract

Streaming data analysis is increasingly required in applications, e.g., IoT, cybersecurity, robotics, mechatronics or cyber-physical systems. Despite its relevance, it is still an emerging field with open challenges. SDO is a recent anomaly detection method designed to meet requirements of speed, interpretability and intuitive parameterization. In this work, we present SDOoop, which extends the capabilities of SDO's streaming version to retain temporal information of data structures. SDOoop spots contextual anomalies undetectable by traditional algorithms, while enabling the inspection of data geometries, clusters and temporal patterns. We used SDOoop to model real network communications in critical infrastructures and extract patterns that disclose their dynamics. Moreover, we evaluated SDOoop with data from intrusion detection and natural science domains and obtained performances equivalent or superior to state-of-the-art approaches. Our results show the high potential of new model-based methods to analyze and explain streaming data. Since SDOoop operates with constant per-sample space and time complexity, it is ideal for big data, being able to instantly process large volumes of information. SDOoop conforms to next-generation machine learning, which, in addition to accuracy and speed, is expected to provide highly interpretable and informative models.