Optimal Lower Bound for Itemset Frequency Indicator Sketches

TL;DR

This paper establishes a tight lower bound on the space complexity for itemset frequency indicator sketches, matching known upper bounds and advancing understanding of data summarization efficiency.

Contribution

It improves the lower bound on the space needed for itemset frequency sketches, specifically for pairs, aligning with existing upper bounds.

Findings

Lower bound of Ω(1/ε * d * log(εd)) bits for k=2

Matching upper bounds for ε ≥ 1/d^{0.99}

Bounds are tight for small ε, including 1/d

Abstract

Given a database, a common problem is to find the pairs or $k$-tuples of items that frequently co-occur. One specific problem is to create a small space "sketch" of the data that records which $k$-tuples appear in more than an $\epsilon$ fraction of rows of the database. We improve the lower bound of Liberty, Mitzenmacher, and Thaler [LMT14], showing that $\Omega(\frac{1}{\epsilon}d \log (\epsilon d))$ bits are necessary even in the case of $k=2$. This matches the sampling upper bound for all $\epsilon \geq 1/d^{.99}$, and (in the case of $k=2$) another trivial upper bound for $\epsilon = 1/d$.