HyperLogLogLog: Cardinality Estimation With One Log More

TL;DR

HyperLogLogLog introduces a highly compressed, mergeable sketch for cardinality estimation that reduces space usage by about 40% while maintaining estimation accuracy and efficiency.

Contribution

It presents a novel compression method for HyperLogLog sketches, significantly reducing space while preserving all estimation and merge properties.

Findings

Achieved approximately 40% reduction in sketch size.

Maintains competitive update and merge times.

Provides a practical implementation with experimental validation.

Abstract

We present HyperLogLogLog, a practical compression of the HyperLogLog sketch that compresses the sketch from $O(m\log\log n)$ bits down to $m \log_2\log_2\log_2 m + O(m+\log\log n)$ bits for estimating the number of distinct elements~$n$ using $m$~registers. The algorithm works as a drop-in replacement that preserves all estimation properties of the HyperLogLog sketch, it is possible to convert back and forth between the compressed and uncompressed representations, and the compressed sketch maintains mergeability in the compressed domain. The compressed sketch can be updated in amortized constant time, assuming $n$ is sufficiently larger than $m$. We provide a C++ implementation of the sketch, and show by experimental evaluation against well-known implementations by Google and Apache that our implementation provides small sketches while maintaining competitive update and merge times. Concretely, we observed approximately a 40% reduction in the sketch size. Furthermore, we obtain as a corollary a theoretical algorithm that compresses the sketch down to $m\log_2\log_2\log_2\log_2 m+O(m\log\log\log m/\log\log m+\log\log n)$ bits.