Learning-based Sketches for Frequency Estimation in Data Streams without Ground Truth

TL;DR

UCL-sketch is a novel online learning-based sketching method for frequency estimation in data streams that requires no ground truth, offers high accuracy, scalability, and significantly faster decoding compared to existing approaches.

Contribution

We introduce UCL-sketch, a scalable, online learning-based frequency estimation method that operates without ground truth and achieves superior accuracy and speed.

Findings

Outperforms previous methods in accuracy and distribution.

Achieves near-oracle quality under tight memory constraints.

Decoding speed is nearly 500 times faster than existing equation-based sketches.

Abstract

Estimating the frequency of items on the high-volume, fast data stream has been extensively studied in many areas, such as database and network measurement. Traditional sketches provide only coarse estimates under strict memory constraints. Although some learning-augmented methods have emerged recently, they typically rely on offline training with real frequencies or/and labels, which are often unavailable. Moreover, these methods suffer from slow update speeds, limiting their suitability for real-time processing despite offering only marginal accuracy improvements. To overcome these challenges, we propose UCL-sketch, a practical learning-based paradigm for per-key frequency estimation. Our design introduces two key innovations: (i) an online training mechanism based on equivalent learning that requires no ground truth (GT), and (ii) a highly scalable architecture leveraging logically structured estimation buckets to scale to real-world data stream. The UCL-sketch, which utilizes compressive sensing (CS), converges to an estimator that provably yields a error bound far lower than that of prior works, without sacrificing the speed of processing. Extensive experiments on both real-world and synthetic datasets demonstrate that our approach outperforms previously proposed approaches regarding per-key accuracy and distribution. Notably, under extremely tight memory budgets, its quality almost matches that of an (infeasible) omniscient oracle. Moreover, compared to the existing equation-based sketch, UCL-sketch achieves an average decoding speedup of nearly 500 times. To help further research and development, our code is publicly available at https://github.com/Y-debug-sys/UCL-sketch.