Flattened Exponential Histogram for Sliding Window Queries over Data Streams

TL;DR

This paper introduces the flattened exponential histogram (FEH), a new data structure for sliding window counting in data streams that offers significantly improved accuracy over existing exponential histograms with similar memory use.

Contribution

The FEH model enhances the exponential histogram technique, providing better accuracy and comparable speed for sliding window counting in data streams.

Findings

FEH achieves 4 to 15 times better accuracy than exponential histogram.

FEH maintains similar speed to existing methods.

Experimental results on real datasets validate FEH's efficiency.

Abstract

The Basic Counting problem [1] is one of the most fundamental and critical streaming problems of sliding window queries over data streams. Given a stream of 0's and 1's, the purpose of this problem is to estimate the number of 1's in the last N elements (or time units) seen from the stream. Its solution can be used as building blocks to solve numerous more complex problems such as heavy hitter, frequency estimation, distinct counting, etc. In this paper, we present the flattened exponential histogram (FEH) model for the Basic Counting problem. Our model improves over the exponential histogram [1], [2], a well-received deterministic technique for Basic Counting problem, with respect to accuracy and memory utilization most of the time in practice. Extensive experimental results on real-world datasets show that with the same memory footprint, the accuracy of our model is between 4 to 15 and on average 7 times better than that of the exponential histogram, while the speed is roughly the same.