Simpler and Better Cardinality Estimators for HyperLogLog and PCSA

TL;DR

This paper introduces a new class of estimators called GRA{} for cardinality estimation in sketching algorithms, which are simpler to compute, more accurate, and nearly optimal compared to existing methods like HyperLogLog and PCSA.

Contribution

The paper defines GRA{} estimators, analyzes their variance, and demonstrates that fractional parameter choices significantly improve accuracy over standard estimators.

Findings

GRA{} estimators closely approach Cramér-Rao bounds.

Fractional au values improve estimator accuracy.

GRA{} estimators are simple to compute and update.

Abstract

\emph{Cardinality Estimation} (aka \emph{Distinct Elements}) is a classic problem in sketching with many industrial applications. Although sketching \emph{algorithms} are fairly simple, analyzing the cardinality \emph{estimators} is notoriously difficult, and even today the state-of-the-art sketches such as HyperLogLog and (compressed) \PCSA{} are not covered in graduate level Big Data courses. In this paper we define a class of \emph{generalized remaining area} (\tGRA) estimators, and observe that HyperLogLog, LogLog, and some estimators for PCSA are merely instantiations of \tGRA{} for various integral values of $\tau$. We then analyze the limiting relative variance of \tGRA{} estimators. It turns out that the standard estimators for HyperLogLog and PCSA can be improved by choosing a \emph{fractional} value of $\tau$. The resulting estimators come \emph{very} close to the Cram\'{e}r-Rao lower bounds for HyperLogLog{} and PCSA derived from their Fisher information. Although the Cram\'{e}r-Rao lower bound \emph{can} be achieved with the Maximum Likelihood Estimator (MLE), the MLE is cumbersome to compute and dynamically update. In contrast, \tGRA{} estimators are trivial to update in constant time. Our presentation assumes only basic calculus and probability, not any complex analysis~\cite{FlajoletM85,DurandF03,FlajoletFGM07}.