DynamicLogLog: Faster, Smaller, and More Accurate Cardinality Estimation

TL;DR

DynamicLogLog (DLL) improves cardinality estimation by reducing memory, increasing speed, and smoothing error profiles compared to HyperLogLog, through shared exponents and hybrid techniques.

Contribution

Introduces DynamicLogLog, a novel cardinality estimator that uses fewer bits, offers faster performance, and achieves more accurate and stable estimates than existing methods.

Findings

DLL reduces memory by 33% compared to HLL.

DLL is over 10x faster than HLL in bandwidth-constrained scenarios.

DLL achieves comparable or better accuracy with less memory.

Abstract

Cardinality estimation - calculating the number of distinct elements in a stream - is a longstanding problem with applications from networking to bioinformatics. HyperLogLog (HLL), the prevailing standard, has a well-known error spike in its transition region and requires 6 bits per bucket, with data structure size scaling as B*log(log(cardinality)). We present DynamicLogLog (DLL), which uses a shared exponent across all buckets, storing only relative leading-zero counts. This yields three benefits: (1) only 4 bits per bucket (33% memory reduction), (2) an early exit mask that rejects >99.9% of elements at high cardinality before any bucket access (over 10x faster than HLL when bandwidth-constrained), and (3) a flat error profile via Dynamic Linear Counting (DLC) and a Logarithmic Hybrid Blend that eliminates HLL's transition artifact. Squaring the maximum representable cardinality requires only a single additional bit of global state. At 2,048 buckets with 512k simulations, DLL4's hybrid estimate achieves 1.830% mean and 1.834% peak absolute error using 1,024 bytes, compared to 1.84% mean and 34.1% peak for HLL using 1,536 bytes. DLC achieves 1.90% mean without correction factors. DynamicUltraLogLog (UDLL6), a fusion of DLL and UltraLogLog, achieves ULL-level accuracy at 75% of the memory. History-corrected variants (Hybrid+n) and Layered DLC (LDLC) provide further improvements using per-state correction tables and anti-phase error cancellation.