Efficient Subspace Search in Data Streams

TL;DR

This paper introduces SGMRD, a novel method for real-time subspace search in high-dimensional data streams, improving pattern detection and outlier identification by leveraging dependency estimators and bandit-based monitoring.

Contribution

It generalizes subspace search to streaming data, combining dependency estimation and bandit theory for efficient, effective pattern monitoring in high-dimensional, evolving data streams.

Findings

SGMRD outperforms existing methods significantly.

It enhances downstream tasks like outlier detection.

Demonstrated effectiveness on synthetic and real-world data.

Abstract

In the real world, data streams are ubiquitous -- think of network traffic or sensor data. Mining patterns, e.g., outliers or clusters, from such data must take place in real time. This is challenging because (1) streams often have high dimensionality, and (2) the data characteristics may change over time. Existing approaches tend to focus on only one aspect, either high dimensionality or the specifics of the streaming setting. For static data, a common approach to deal with high dimensionality -- known as subspace search -- extracts low-dimensional, `interesting' projections (subspaces), in which patterns are easier to find. In this paper, we address both Challenge (1) and (2) by generalising subspace search to data streams. Our approach, Streaming Greedy Maximum Random Deviation (SGMRD), monitors interesting subspaces in high-dimensional data streams. It leverages novel multivariate dependency estimators and monitoring techniques based on bandit theory. We show that the benefits of SGMRD are twofold: (i) It monitors subspaces efficiently, and (ii) this improves the results of downstream data mining tasks, such as outlier detection. Our experiments, performed against synthetic and real-world data, demonstrate that SGMRD outperforms its competitors by a large margin.