MaxSketch: Robust Distinct Counting in Streams via Random Projections

TL;DR

MaxSketch introduces a simple, Gaussian projection-based sketch that efficiently estimates the number of distinct high-dimensional, noisy objects in data streams by leveraging geometric structure, outperforming classical methods.

Contribution

The paper presents MaxSketch, a novel max-linear sketch that exploits geometric structure in learned representations to significantly improve memory efficiency for distinct counting.

Findings

MaxSketch accurately estimates distinct counts in image streams.

It requires only logarithmic in n, inverse squared epsilon, number of projections for (1+epsilon) accuracy.

Experiments show MaxSketch outperforms classical methods and generalizes beyond training data.

Abstract

Estimating the number of distinct elements in a data stream is well understood when repeated elements are identical. In modern settings, however, observations are high-dimensional and noisy, so repeated instances of the same object are only approximately similar -- for example, different images of the same individual may vary significantly at the pixel level. Classical sketches such as HyperLogLog rely on consistent hash values for identical elements and break down in this regime. Recent work on robust distinct counting in general metric spaces achieves $\widetilde{\Theta}(\sqrt{n})$ memory, which is tight in the worst case. We show that substantially improved memory guarantees are possible under geometric structure common in learned representations. We introduce MaxSketch, a simple max-linear sketch built from random Gaussian projections, and prove that it succeeds in estimating the number of distinct latent objects. Concretely, we show that under this assumption $m = \widetilde{O} (\log n / \varepsilon^2)$ random projections (and hence $\widetilde{O} (\log n/\varepsilon^2)$ memory) suffice to recover the true distinct count within a $(1+\varepsilon)$ factor. Experiments on image streams confirm that MaxSketch accurately estimates distinct counts and generalizes beyond the training regime. Our results bridge classical streaming algorithms and modern representation learning, showing how geometric structure can fundamentally reduce the complexity of distinct counting.