Optimal streaming and tracking distinct elements with high probability

TL;DR

This paper improves the space complexity bounds for streaming algorithms estimating the number of distinct elements, removing unnecessary multiplicative factors and providing optimal solutions for both standard and strong tracking variants.

Contribution

It presents an optimal streaming algorithm for distinct elements that avoids space blow-up from multiple repetitions and establishes tight bounds for strong tracking.

Findings

Optimal space complexity for approximate distinct count without multiplicative blow-up.

Space bounds for strong tracking variant are proven to be optimal.

The results settle the space complexity for all standard parameters.

Abstract

The distinct elements problem is one of the fundamental problems in streaming algorithms --- given a stream of integers in the range $\{1,\ldots,n\}$, we wish to provide a $(1+\varepsilon)$ approximation to the number of distinct elements in the input. After a long line of research an optimal solution for this problem with constant probability of success, using $\mathcal{O}(\frac{1}{\varepsilon^2}+\log n)$ bits of space, was given by Kane, Nelson and Woodruff in 2010. The standard approach used in order to achieve low failure probability $\delta$ is to take the median of $\log \delta^{-1}$ parallel repetitions of the original algorithm. We show that such a multiplicative space blow-up is unnecessary: we provide an optimal algorithm using $\mathcal{O}(\frac{\log \delta^{-1}}{\varepsilon^2} + \log n)$ bits of space --- matching known lower bounds for this problem. That is, the $\log\delta^{-1}$ factor does not multiply the $\log n$ term. This settles completely the space complexity of the distinct elements problem with respect to all standard parameters. We consider also the \emph{strong tracking} (or \emph{continuous monitoring}) variant of the distinct elements problem, where we want an algorithm which provides an approximation of the number of distinct elements seen so far, at all times of the stream. We show that this variant can be solved using $\mathcal{O}(\frac{\log \log n + \log \delta^{-1}}{\varepsilon^2} + \log n)$ bits of space, which we show to be optimal.