A Framework for Estimating Stream Expression Cardinalities

TL;DR

This paper introduces a comprehensive framework for accurately estimating the number of unique elements in complex set expressions over multiple distributed data streams, unifying existing methods and proposing new algorithms.

Contribution

It presents a broad class of unbiased estimators with strong variance bounds, generalizing previous results and introducing novel sampling algorithms with improved tradeoffs.

Findings

Estimators are perfectly unbiased with strong variance bounds.

New sampling algorithms achieve better accuracy-space tradeoffs.

Framework unifies and extends prior work on stream cardinality estimation.

Abstract

Given $m$ distributed data streams $A_1, \dots, A_m$, we consider the problem of estimating the number of unique identifiers in streams defined by set expressions over $A_1, \dots, A_m$. We identify a broad class of algorithms for solving this problem, and show that the estimators output by any algorithm in this class are perfectly unbiased and satisfy strong variance bounds. Our analysis unifies and generalizes a variety of earlier results in the literature. To demonstrate its generality, we describe several novel sampling algorithms in our class, and show that they achieve a novel tradeoff between accuracy, space usage, update speed, and applicability.