Fair-Count-Min: Frequency Estimation under Equal Group-wise Approximation Factor

TL;DR

Fair-Count-Min is a novel frequency estimation sketch that ensures equal approximation fairness across different element groups in streaming data, addressing bias issues inherent in traditional Count-Min sketches.

Contribution

It introduces a group-aware hashing method and theoretical guarantees to achieve fair frequency estimation with minimal error increase.

Findings

Achieves fairness with minimal additional error

Maintains competitive efficiency compared to standard Count-Min sketches

Validated through extensive experiments on real-world and synthetic datasets

Abstract

Frequency estimation in streaming data often relies on sketches like Count-Min (CM) to provide approximate answers with sublinear space. However, CM sketches introduce additive errors that disproportionately impact low-frequency elements, creating fairness concerns across different groups of elements. We introduce Fair-Count-Min, a frequency estimation sketch that guarantees equal expected approximation factors across element groups, thus addressing the unfairness issue. We propose a column partitioning approach with group-aware semi-uniform hashing to eliminate collisions between elements from different groups. We provide theoretical guarantees for fairness, analyze the price of fairness, and validate our theoretical findings through extensive experiments on real-world and synthetic datasets. Our experimental results show that Fair-Count-Min achieves fairness with minimal additional error and maintains competitive efficiency compared to standard CM sketches.