ExtendedHyperLogLog: Analysis of a new Cardinality Estimator

TL;DR

This paper introduces ExtendedHyperLogLog (EHLL), a new probabilistic data structure that improves memory efficiency over HyperLogLog for estimating stream cardinality, including in martingale settings.

Contribution

EHLL reduces memory usage compared to HyperLogLog while maintaining similar accuracy, and extends applicability to martingale-based estimations.

Findings

EHLL uses 16% less memory than HLL for same accuracy.

Martingale EHLL achieves same accuracy as Martingale HLL with 12% less memory.

EHLL is effective in both standard and martingale estimation frameworks.

Abstract

We discuss the problem of counting distinct elements in a stream. A stream is usually considered as a sequence of elements that come one at a time. An exact solution to the problem requires memory space of the size of the stream. For many applications this solution is infeasible due to very large streams. The solution in that case, is to use a probabilistic data structure (also called sketch), from which we can estimate with high accuracy the cardinality of the stream. We present a new algorithm, ExtendedHyperLogLog (EHLL), which is based on the state-of-the-art algorithm, HyperLogLog (HLL). In order to achieve the same accuracy as HLL, EHLL uses 16% less memory. In recent years, a martingale approach has bean developed. In the martingale setting we receive better accuracy at the price of not being able to merge sketches. EHLL also works in the martingale setting. Martingale EHLL achieves the same accuracy as Martingale HLL using 12% less memory.