Revisiting Frequency Moment Estimation in Random Order Streams

TL;DR

This paper demonstrates that in the random-order data stream model, frequency moment estimation can be significantly more space-efficient than in the turnstile model, achieving near-optimal deterministic algorithms with substantially reduced space complexity.

Contribution

The authors show that in the random-order model, frequency moment estimation can be done with much less space, and they provide a deterministic algorithm with near-optimal bounds.

Findings

Achieves $ ilde{O}( ext{epsilon}^{-2} + ext{log} n)$ space for frequency moments in random-order streams.

Provides a deterministic algorithm with near-optimal space bounds in the random-order model.

Shows quadratic improvement over turnstile model when $ ext{epsilon}^{-2} oughly ext{log} n$.

Abstract

We revisit one of the classic problems in the data stream literature, namely, that of estimating the frequency moments $F_p$ for $0 < p < 2$ of an underlying $n$-dimensional vector presented as a sequence of additive updates in a stream. It is well-known that using $p$-stable distributions one can approximate any of these moments up to a multiplicative $(1+\epsilon)$-factor using $O(\epsilon^{-2} \log n)$ bits of space, and this space bound is optimal up to a constant factor in the turnstile streaming model. We show that surprisingly, if one instead considers the popular random-order model of insertion-only streams, in which the updates to the underlying vector arrive in a random order, then one can beat this space bound and achieve $\tilde{O}(\epsilon^{-2} + \log n)$ bits of space, where the $\tilde{O}$ hides poly$(\log(1/\epsilon) + \log \log n)$ factors. If $\epsilon^{-2} \approx \log n$, this represents a roughly quadratic improvement in the space achievable in turnstile streams. Our algorithm is in fact deterministic, and we show our space bound is optimal up to poly$(\log(1/\epsilon) + \log \log n)$ factors for deterministic algorithms in the random order model. We also obtain a similar improvement in space for $p = 2$ whenever $F_2 \gtrsim \log n\cdot F_1$.