On the k-Independence Required by Linear Probing and Minwise Independence

TL;DR

This paper establishes the independence requirements for linear probing and minwise independence, demonstrating that higher independence levels are necessary for expected constant-time performance and approximate minwise independence.

Contribution

It proves that 5-independence is needed for linear probing's efficiency and that \\Omega(log 1/\\epsilon)-independence is required for approximate minwise independence, clarifying theoretical bounds.

Findings

4-independent hash functions lead to logarithmic search time in linear probing.

Higher independence levels are necessary for optimal performance in hashing schemes.

The 2-independent multiply-shift scheme performs poorly in these applications.

Abstract

We show that linear probing requires 5-independent hash functions for expected constant-time performance, matching an upper bound of [Pagh et al. STOC'07]. More precisely, we construct a 4-independent hash functions yielding expected logarithmic search time. For (1+{\epsilon})-approximate minwise independence, we show that \Omega(log 1/{\epsilon})-independent hash functions are required, matching an upper bound of [Indyk, SODA'99]. We also show that the very fast 2-independent multiply-shift scheme of Dietzfelbinger [STACS'96] fails badly in both applications.