LMQ-Sketch: Lagom Multi-Query Sketch for High-Rate Online Analytics

TL;DR

LMQ-Sketch introduces a unified, high-throughput data sketch that efficiently supports multiple concurrent queries and updates in high-rate data streams, with strong accuracy and low memory use.

Contribution

It presents LMQ-Sketch, a novel concurrent multi-query sketch supporting multiple query types with low latency, high throughput, and reduced memory footprint, addressing key challenges in high-rate data analytics.

Findings

Supports multiple query types concurrently with updates

Achieves low-latency global querying (<100 us)

Reduces memory usage by an order of magnitude compared to prior methods

Abstract

Data sketches balance resource efficiency with controllable approximations for extracting features in high-volume, high-rate data. Two important points of interest are highlighted separately in recent works; namely, to (1) answer multiple types of queries from one pass, and (2) query concurrently with updates. Several fundamental challenges arise when integrating these directions, which we tackle in this work. We investigate the trade-offs to be balanced and synthesize key ideas into LMQ-Sketch, a single, composite data sketch supporting multiple queries (frequency point queries, frequency moments F1, and F2) concurrently with updates. Our method 'Lagom' is a cornerstone of LMQ-Sketch for low-latency global querying (<100 us), combining freshness, timeliness, and accuracy with a low memory footprint and high throughput (>2B updates/s). We analyze and evaluate the accuracy of Lagom, which builds on a simple geometric argument and efficiently combines work distribution with synchronization for proper concurrency semantics -- monotonicity of operations and intermediate value linearizability. Comparing with state-of-the-art methods (which, as mentioned, only cover either mixed queries or concurrency), LMQ-Sketch shows highly competitive throughput, with additional accuracy guarantees and concurrency semantics, while also reducing the required memory budget by an order of magnitude. We expect the methodology to have broader impact on concurrent multi-query sketches.