Combined Search and Encoding for Seeds, with an Application to Minimal Perfect Hashing

TL;DR

This paper introduces a space-efficient algorithm for storing seeds in randomized algorithms, reducing entropy and improving minimal perfect hash function construction with significantly faster throughput.

Contribution

It presents a novel method to encode successful seeds more efficiently, lowering entropy and enabling near-optimal space usage in minimal perfect hashing.

Findings

Reduces seed storage entropy by Ω(n) bits in most cases.

Achieves (1+ε)OPT space with O(n/ε) construction time.

Outperforms state-of-the-art in construction throughput by over 100x for small ε.

Abstract

Randomised algorithms often employ methods that can fail and that are retried with independent randomness until they succeed. Randomised data structures therefore often store indices of successful attempts, called seeds. If $n$ such seeds are required (e.g., for independent substructures) the standard approach is to compute for each $i \in [n]$ the smallest successful seed $S_i$ and store $\vec{S} = (S_1, \ldots, S_n)$. The central observation of this paper is that this is not space-optimal. We present a different algorithm that computes a sequence $\vec{S}' = (S_1', \ldots, S_n')$ of successful seeds such that the entropy of $\vec{S'}$ undercuts the entropy of $\vec{S}$ by $\Omega(n)$ bits in most cases. To achieve a memory consumption of $\mathrm{OPT}+\varepsilon n$, the expected number of inspected seeds increases by a factor of $O(1/\varepsilon)$. We demonstrate the usefulness of our findings with a novel construction for minimal perfect hash functions that, for $n$ keys and any $\varepsilon \in [n^{-3/7}, 1]$, has space requirement $(1+\varepsilon)\mathrm{OPT}$ and construction time $O(n/\varepsilon)$. All previous approaches only support $\varepsilon = \omega(1 / \log n)$ or have construction times that increase exponentially with $1/\varepsilon$. Our implementation beats the construction throughput of the state of the art by more than two orders of magnitude for $\varepsilon \leq 3\%$.