LMQ-Sketch: Lagom Multi-Query Sketch for High-Rate Online Analytics
Martin Hilgendorf, Marina Papatriantafilou

TL;DR
LMQ-Sketch introduces a unified, high-throughput data sketch that efficiently supports multiple concurrent queries and updates in high-rate data streams, with strong accuracy and low memory use.
Contribution
It presents LMQ-Sketch, a novel concurrent multi-query sketch supporting multiple query types with low latency, high throughput, and reduced memory footprint, addressing key challenges in high-rate data analytics.
Findings
Supports multiple query types concurrently with updates
Achieves low-latency global querying (<100 us)
Reduces memory usage by an order of magnitude compared to prior methods
Abstract
Data sketches balance resource efficiency with controllable approximations for extracting features in high-volume, high-rate data. Two important points of interest are highlighted separately in recent works; namely, to (1) answer multiple types of queries from one pass, and (2) query concurrently with updates. Several fundamental challenges arise when integrating these directions, which we tackle in this work. We investigate the trade-offs to be balanced and synthesize key ideas into LMQ-Sketch, a single, composite data sketch supporting multiple queries (frequency point queries, frequency moments F1, and F2) concurrently with updates. Our method 'Lagom' is a cornerstone of LMQ-Sketch for low-latency global querying (<100 us), combining freshness, timeliness, and accuracy with a low memory footprint and high throughput (>2B updates/s). We analyze and evaluate the accuracy of Lagom,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
