Cardinality estimation using Gumbel distribution

TL;DR

This paper introduces a Gumbel distribution-based modification to LogLog and HyperLogLog algorithms, simplifying their analysis and improving estimator smoothness for cardinality estimation in large datasets.

Contribution

It proposes a novel Gumbel distribution approach that simplifies analysis and enhances the performance of existing cardinality estimation algorithms.

Findings

Simpler, more elementary analysis of estimators

Smoother estimator behavior

Potential improvements in estimation accuracy

Abstract

Cardinality estimation is the task of approximating the number of distinct elements in a large dataset with possibly repeating elements. LogLog and HyperLogLog (c.f. Durand and Flajolet [ESA 2003], Flajolet et al. [Discrete Math Theor. 2007]) are small space sketching schemes for cardinality estimation, which have both strong theoretical guarantees of performance and are highly effective in practice. This makes them a highly popular solution with many implementations in big-data systems (e.g. Algebird, Apache DataSketches, BigQuery, Presto and Redis). However, despite having simple and elegant formulation, both the analysis of LogLog and HyperLogLog are extremely involved -- spanning over tens of pages of analytic combinatorics and complex function analysis. We propose a modification to both LogLog and HyperLogLog that replaces discrete geometric distribution with a continuous Gumbel distribution. This leads to a very short, simple and elementary analysis of estimation guarantees, and smoother behavior of the estimator.