Quancurrent: A Concurrent Quantiles Sketch

TL;DR

Quancurrent is a highly scalable concurrent Quantiles sketch that significantly improves throughput and query freshness in streaming data analytics by enabling parallel updates and queries.

Contribution

It introduces Quancurrent, a novel concurrent Quantiles sketch that scales linearly with threads and enhances real-time data distribution estimation.

Findings

Linear throughput increase with threads

12x update speedup with 32 threads

30x query speedup with 32 threads

Abstract

Sketches are a family of streaming algorithms widely used in the world of big data to perform fast, real-time analytics. A popular sketch type is Quantiles, which estimates the data distribution of a large input stream. We present Quancurrent, a highly scalable concurrent Quantiles sketch. Quancurrent's throughput increases linearly with the number of available threads, and with $32$ threads, it reaches an update speedup of $12$x and a query speedup of $30$x over a sequential sketch. Quancurrent allows queries to occur concurrently with updates and achieves an order of magnitude better query freshness than existing scalable solutions.