Frugal Streaming for Estimating Quantiles:One (or two) memory suffices
Qiang Ma, S. Muthukrishnan, Mark Sandler

TL;DR
This paper introduces frugal streaming algorithms that estimate quantiles using minimal memory per group, enabling efficient processing of large-scale data streams with strict memory constraints.
Contribution
The authors present the first single-memory-unit algorithm for quantile estimation in streaming data, along with an extension using two units, demonstrating efficiency and stability in practical scenarios.
Findings
Single-memory-unit algorithm accurately estimates quantiles.
Two-memory-unit extension converges faster.
Algorithms perform comparably to existing methods with less memory.
Abstract
Modern applications require processing streams of data for estimating statistical quantities such as quantiles with small amount of memory. In many such applications, in fact, one needs to compute such statistical quantities for each of a large number of groups, which additionally restricts the amount of memory available for the stream for any particular group. We address this challenge and introduce frugal streaming, that is algorithms that work with tiny -- typically, sub-streaming -- amount of memory per group. We design a frugal algorithm that uses only one unit of memory per group to compute a quantile for each group. For stochastic streams where data items are drawn from a distribution independently, we analyze and show that the algorithm finds an approximation to the quantile rapidly and remains stably close to it. We also propose an extension of this algorithm that uses two…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Stream Mining Techniques · Advanced Bandit Algorithms Research · Machine Learning and Data Classification
