On Fine-Grained Distinct Element Estimation

TL;DR

This paper introduces a new parameterization for distributed distinct element estimation, leading to more efficient algorithms that outperform previous bounds when the number of collisions is small, and provides matching lower bounds.

Contribution

The authors propose a collision-based parameterization and develop algorithms with improved communication complexity, establishing $C$ as a tight measure and extending to streaming scenarios.

Findings

New collision-based parameterization improves efficiency when collisions are few.

Algorithms match lower bounds across different regimes, confirming tightness.

Extensions to streaming algorithms for high-frequency items.

Abstract

We study the problem of distributed distinct element estimation, where $\alpha$ servers each receive a subset of a universe $[n]$ and aim to compute a $(1+\varepsilon)$-approximation to the number of distinct elements using minimal communication. While prior work establishes a worst-case bound of $\Theta\left(\alpha\log n+\frac{\alpha}{\varepsilon^2}\right)$ bits, these results rely on assumptions that may not hold in practice. We introduce a new parameterization based on the number $C = \frac{\beta}{\varepsilon^2}$ of pairwise collisions, i.e., instances where the same element appears on multiple servers, and design a protocol that uses only $\mathcal{O}\left(\alpha\log n+\frac{\sqrt{\beta}}{\varepsilon^2} \log n\right)$ bits, breaking previous lower bounds when $C$ is small. We further improve our algorithm under assumptions on the number of distinct elements or collisions and provide matching lower bounds in all regimes, establishing $C$ as a tight complexity measure for the problem. Finally, we consider streaming algorithms for distinct element estimation parameterized by the number of items with frequency larger than $1$. Overall, our results offer insight into why statistical problems with known hardness results can be efficiently solved in practice.