MagnifierSketch: Quantile Estimation Centered at One Point

TL;DR

MagnifierSketch is a novel algorithm designed for accurate, high-throughput quantile estimation centered at a specific point in data streams, improving accuracy and efficiency over existing methods.

Contribution

It introduces MagnifierSketch, a new method for point-quantile estimation supporting both single-key and per-key scenarios with proven unbiasedness and optimized complexity.

Findings

Significantly lower average error than state-of-the-art methods.

Supports high-speed data streams with improved throughput.

Reduces quantile query latency in real database systems.

Abstract

In this paper, we take into consideration quantile estimation in data stream models, where every item in the data stream is a key-value pair. Researchers sometimes aim to estimate per-key quantiles (i.e. quantile estimation for every distinct key), and some popular use cases, such as tail latency measurement, recline on a predefined single quantile (e.g. 0.95- or 0.99- quantile) rather than demanding arbitrary quantile estimation. However, existing algorithms are not specially designed for per-key estimation centered at one point. They cannot achieve high accuracy in our problem setting, and their throughput are not satisfactory to handle high-speed items in data streams. To solve this problem, we propose MagnifierSketch for point-quantile estimation. MagnifierSketch supports both single-key and per-key quantile estimation, and its key techniques are named Value Focus, Distribution Calibration and Double Filtration. We provide strict mathematical derivations to prove the unbiasedness of MagnifierSketch and show its space and time complexity. Our experimental results show that the Average Error (AE) of MagnifierSketch is significantly lower than the state-of-the-art in both single-key and per-key situations. We also implement MagnifierSketch on RocksDB database to reduce quantile query latency in real databases. All related codes of MagnifierSketch are open-sourced and available at GitHub.