HyperLogLog Sketch Acceleration on FPGA
Amit Kulkarni, Monica Chiosa, Thomas B. Preu{\ss}er, Kaan Kara, David, Sidler, Gustavo Alonso

TL;DR
This paper presents an FPGA implementation of HyperLogLog that significantly accelerates cardinality estimation in high-speed data streams, achieving 1.8 times higher throughput than a high-end CPU system.
Contribution
The paper introduces a parallel FPGA-based HyperLogLog implementation that outperforms traditional CPU-based solutions in processing high-speed data streams.
Findings
Achieved 1.8x higher throughput than CPU implementation
Demonstrated effective FPGA parallelism for data sketching
Enhanced real-time data stream analysis capabilities
Abstract
Data sketches are a set of widely used approximated data summarizing techniques. Their fundamental property is sub-linear memory complexity on the input cardinality, an important aspect when processing streams or data sets with a vast base domain (URLs, IP addresses, user IDs, etc.). Among the many data sketches available, HyperLogLog has become the reference for cardinality counting (how many distinct data items there are in a data set). Although it does not count every data item (to reduce memory consumption), it provides probabilistic guarantees on the result, and it is, thus, often used to analyze data streams. In this paper, we explore how to implement HyperLogLog on an FPGA to benefit from the parallelism available and the ability to process data streams coming from high-speed networks. Our multi-pipelined high-cardinality HyperLogLog implementation delivers 1.8x higher throughput…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
