Efficient Algorithm for Deterministic Search of Hot Elements

TL;DR

This paper introduces a new deterministic online algorithm for identifying frequent elements in large data streams, achieving optimal memory and time efficiency without randomness or multiple passes.

Contribution

It presents the first truly online deterministic algorithm for frequent element detection with near-optimal scalability and memory usage, improving over prior randomized or multi-pass methods.

Findings

Uses $O( ext{min}(n, rac{ ext{polylog}(n)}{ ext{epsilon}}))$ memory

Operates in $O( ext{polylog}(n))$ time per element

Establishes a lower bound of $ ext{Omega}( ext{min}(n, rac{1}{ ext{epsilon}}))$ on memory requirements

Abstract

When facing a very large stream of data, it is often desirable to extract most important statistics online in a short time and using small memory. For example, one may want to quickly find the most influential users generating posts online or check if the stream contains many identical elements. In this paper, we study streams containing insertions and deletions of elements from a possibly large set $N$ of size $|N| = n$, that are being processed by online deterministic algorithms. At any point in the stream the algorithm may be queried to output elements of certain frequency in the already processed stream. More precisely, the most frequent elements in the stream so far. The output is considered correct if the returned elements it contains all elements with frequency greater than a given parameter $\varphi$ and no element with frequency smaller than $\varphi-\epsilon$. We present an efficient online deterministic algorithm for solving this problem using $O(\min(n, \frac{polylog(n)}{\epsilon}))$ memory and $O(polylog(n))$ time per processing and outputting an element. It is the first such algorithm as the previous algorithms were either randomized, or processed elements in substantially larger time $\Omega(\min(n, \frac{\log n}{\epsilon}))$, or handled only insertions and required two passes over the stream (i.e., were not truly online). Our solution is almost-optimally scalable (with only a polylogarithmic overhead) and does not require randomness or scanning twice through the stream. We complement the algorithm analysis with a lower bound $\Omega(\min(n, \frac{1}{\epsilon}))$ on required memory.