Polynomial Estimators for High Frequency Moments

TL;DR

This paper introduces a space-efficient algorithm for estimating high frequency moments in data streams, improving upon previous methods by using novel polynomial estimators and achieving lower space complexity for certain ranges of p.

Contribution

The paper presents a new polynomial-based estimator technique for high frequency moments, reducing space complexity compared to prior algorithms for 2 < p < log(n).

Findings

Achieves space complexity improvements over previous algorithms.

Uses a novel Taylor polynomial-based estimator technique.

Provides bounds on space and update time for the algorithm.

Abstract

We present an algorithm for computing $F_p$, the $p$th moment of an $n$-dimensional frequency vector of a data stream, for $2 < p < \log (n) $, to within $1\pm \epsilon$ factors, $\epsilon \in [n^{-1/p},1]$ with high constant probability. Let $m$ be the number of stream records and $M$ be the largest magnitude of a stream update. The algorithm uses space in bits $$ O(p^2\epsilon^{-2}n^{1-2/p}E(p,n) \log (n) \log (nmM)/\min(\log (n),\epsilon^{4/p-2}))$$ where, $E(p,n) = (1-2/p)^{-1}(1-n^{-4(1-2/p})$. Here $E(p,n)$ is $ O(1)$ for $p = 2+\Omega(1)$ and $ O(\log n)$ for $p = 2 + O(1/\log (n)$. This improves upon the space required by current algorithms \cite{iw:stoc05,bgks:soda06,ako:arxiv10,bo:arxiv10} by a factor of at least $\Omega(\epsilon^{-4/p} \min(\log (n), \epsilon^{4/p-2}))$. The update time is $O(\log (n))$. We use a new technique for designing estimators for functions of the form $\psi(\expect{X})$, where, $X$ is a random variable and $\psi$ is a smooth function, based on a low-degree Taylor polynomial expansion of $\psi(\expect{X})$ around an estimate of $\expect{X}$.