Frequent Elements with Witnesses in Data Streams

TL;DR

This paper extends the classic frequent elements problem in data streams to include associated meta-data, providing optimal algorithms for both insertion-only and insertion-deletion models with space complexity bounds.

Contribution

It introduces the witness version of the frequent elements problem, incorporating meta-data, and presents provably optimal algorithms with tight space bounds for both stream models.

Findings

Optimal space complexity algorithms for insertion-only streams.

Optimal space bounds for insertion-deletion streams.

Extension of frequent elements detection to include meta-data.

Abstract

Detecting frequent elements is among the oldest and most-studied problems in the area of data streams. Given a stream of $m$ data items in $\{1, 2, \dots, n\}$, the objective is to output items that appear at least $d$ times, for some threshold parameter $d$, and provably optimal algorithms are known today. However, in many applications, knowing only the frequent elements themselves is not enough: For example, an Internet router may not only need to know the most frequent destination IP addresses of forwarded packages, but also the timestamps of when these packages appeared or any other meta-data that "arrived" with the packages, e.g., their source IP addresses. In this paper, we introduce the witness version of the frequent elements problem: Given a desired approximation guarantee $\alpha \ge 1$ and a desired frequency $d \le \Delta$, where $\Delta$ is the frequency of the most frequent item, the objective is to report an item together with at least $d / \alpha$ timestamps of when the item appeared in the stream (or any other meta-data that arrived with the items). We give provably optimal algorithms for both the insertion-only and insertion-deletion stream settings: In insertion-only streams, we show that space $\tilde{O}(n + d \cdot n^{\frac{1}{\alpha}})$ is necessary and sufficient for every integral $1 \le \alpha \le \log n$. In insertion-deletion streams, we show that space $\tilde{O}(\frac{n \cdot d}{\alpha^2})$ is necessary and sufficient, for every $\alpha \le \sqrt{n}$.