A randomized online quantile summary in $O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$ words

TL;DR

This paper introduces a new randomized online quantile summary that significantly reduces memory usage to $O(rac{1}{epsilon} ext{log} rac{1}{epsilon})$ words, improving efficiency over previous methods.

Contribution

It presents the first randomized online quantile summary with optimal space complexity matching lower bounds, outperforming deterministic approaches.

Findings

Memory usage is reduced to $O(rac{1}{epsilon} ext{log} rac{1}{epsilon})$ words.

The summary achieves high success probability with minimal space.

It improves upon the previous best upper bound by a factor of $ ext{log}^{1/2} rac{1}{epsilon}$.

Abstract

A quantile summary is a data structure that approximates to $\varepsilon$-relative error the order statistics of a much larger underlying dataset. In this paper we develop a randomized online quantile summary for the cash register data input model and comparison data domain model that uses $O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$ words of memory. This improves upon the previous best upper bound of $O(\frac{1}{\varepsilon} \log^{3/2} \frac{1}{\varepsilon})$ by Agarwal et. al. (PODS 2012). Further, by a lower bound of Hung and Ting (FAW 2010) no deterministic summary for the comparison model can outperform our randomized summary in terms of space complexity. Lastly, our summary has the nice property that $O(\frac{1}{\varepsilon} \log \frac{1}{\varepsilon})$ words suffice to ensure that the success probability is $1 - e^{-\text{poly}(1/\varepsilon)}$.