Detecting Flow Gaps in Data Streams

TL;DR

This paper introduces GapFilter, a novel method for detecting flow gaps in data streams by monitoring value variation, achieving high accuracy and speed with minimal memory usage, and providing two optimized versions for different priorities.

Contribution

First to detect flow gaps in data streams, proposing GapFilter with innovative techniques like similarity absorption and civilian-suspect mechanism, and offering two tailored versions for speed and accuracy.

Findings

GapFilter-AO uses 1/32 memory of baseline for similar accuracy.

GapFilter-SO is 3 times faster than baseline.

Theoretical proof guarantees high accuracy with minimal memory.

Abstract

Data stream monitoring is a crucial task which has a wide range of applications. The majority of existing research in this area can be broadly classified into two types, monitoring value sum and monitoring value cardinality. In this paper, we define a third type, monitoring value variation, which can help us detect flow gaps in data streams. To realize this function, we propose GapFilter, leveraging the idea of Sketch for achieving speed and accuracy. To the best of our knowledge, this is the first work to detect flow gaps in data streams. Two key ideas of our work are the similarity absorption technique and the civilian-suspect mechanism. The similarity absorption technique helps in reducing memory usage and enhancing speed, while the civilian-suspect mechanism further boosts accuracy by organically integrating broad monitoring of overall flows with meticulous monitoring of suspicious flows.We have developed two versions of GapFilter. Speed-Oriented GapFilter (GapFilter-SO) emphasizes speed while maintaining satisfactory accuracy. Accuracy-Oriented GapFilter (GapFilter-AO) prioritizes accuracy while ensuring considerable speed. We provide a theoretical proof demonstrating that GapFilter secures high accuracy with minimal memory usage. Further, extensive experiments were conducted to assess the accuracy and speed of our algorithms. The results reveal that GapFilter-AO requires, on average, 1/32 of the memory to match the accuracy of the Straw-man solution. GapFilter-SO operates at a speed 3 times faster than the Straw-man solution. All associated source code has been open-sourced and is available on GitHub.