Fast Scalable Construction of (Minimal Perfect Hash) Functions

TL;DR

This paper introduces heuristics and programming techniques to significantly speed up the construction of minimal perfect hash functions, making them more practical and space-efficient compared to existing methods.

Contribution

It presents a set of heuristics, broadword programming, and lazy Gaussian elimination to accelerate linear system solving for hash function construction.

Findings

Achieved minimal perfect hash functions with 2.24 bits per element.

Reduced construction time to be competitive with existing hypergraph peeling methods.

Improved space and lookup efficiency for static function data structures.

Abstract

Recent advances in random linear systems on finite fields have paved the way for the construction of constant-time data structures representing static functions and minimal perfect hash functions using less space with respect to existing techniques. The main obstruction for any practical application of these results is the cubic-time Gaussian elimination required to solve these linear systems: despite they can be made very small, the computation is still too slow to be feasible. In this paper we describe in detail a number of heuristics and programming techniques to speed up the resolution of these systems by several orders of magnitude, making the overall construction competitive with the standard and widely used MWHC technique, which is based on hypergraph peeling. In particular, we introduce broadword programming techniques for fast equation manipulation and a lazy Gaussian elimination algorithm. We also describe a number of technical improvements to the data structure which further reduce space usage and improve lookup speed. Our implementation of these techniques yields a minimal perfect hash function data structure occupying 2.24 bits per element, compared to 2.68 for MWHC-based ones, and a static function data structure which reduces the multiplicative overhead from 1.23 to 1.03.