Testing frequency distributions in a stream

TL;DR

This paper introduces methods to verify if a data stream's frequency distribution matches a target distribution using a new distance measure, with efficient algorithms for certain cases and space lower bounds.

Contribution

It proposes the relative Fréchet distance for comparing distributions and develops streaming algorithms for testing distribution closeness under different models.

Findings

Space complexity is Omega(n) for uniform distributions.

Efficient algorithms with O(log^2 n * log log n) space for rapidly decreasing distributions.

The approach combines the Spacesaving algorithm with stream sampling.

Abstract

We study how to verify specific frequency distributions when we observe a stream of $N$ data items taken from a universe of $n$ distinct items. We introduce the \emph{relative Fr\'echet distance} to compare two frequency functions in a homogeneous manner. We consider two streaming models: insertions only and sliding windows. We present a Tester for a certain class of functions, which decides if $f $ is close to $g$ or if $f$ is far from $g$ with high probability, when $f$ is given and $g$ is defined by a stream. If $f$ is uniform we show a space $\Omega(n)$ lower bound. If $f$ decreases fast enough, we then only use space $O(\log^2 n\cdot \log\log n)$. The analysis relies on the Spacesaving algorithm \cite{MAE2005,Z22} and on sampling the stream.