Optimality of Frequency Moment Estimation

TL;DR

This paper establishes the optimal space complexity bounds for estimating the second frequency moment in data streams, and introduces an improved algorithm that matches these bounds for small error parameters.

Contribution

It proves a tight lower bound for frequency moment estimation space complexity and presents a revised algorithm that achieves this bound.

Findings

Lower bound of ig(rac{\u221a{n}\u00b7\u03b5^2}{}ig) bits for all  = (1/)

Revised algorithm matching the lower bound for small 

Extension of bounds to all  = (1/) regimes

Abstract

Estimating the second frequency moment of a stream up to $(1\pm\varepsilon)$ multiplicative error requires at most $O(\log n / \varepsilon^2)$ bits of space, due to a seminal result of Alon, Matias, and Szegedy. It is also known that at least $\Omega(\log n + 1/\varepsilon^{2})$ space is needed. We prove an optimal lower bound of $\Omega\left(\log \left(n \varepsilon^2 \right) / \varepsilon^2\right)$ for all $\varepsilon = \Omega(1/\sqrt{n})$. Note that when $\varepsilon>n^{-1/2 + c}$, where $c>0$, our lower bound matches the classic upper bound of AMS. For smaller values of $\varepsilon$ we also introduce a revised algorithm that improves the classic AMS bound and matches our lower bound.