External-Memory Multimaps

TL;DR

This paper introduces efficient external-memory data structures for multimaps, enabling constant expected I/O performance by combining cuckoo hashing and multiqueues, optimizing key-value pair management in large-scale systems.

Contribution

It presents the first external-memory multimap implementation with constant expected I/O performance using a novel combination of cuckoo hashing and multiqueues.

Findings

Achieves constant expected I/O performance for external-memory multimaps.

Uses a combination of cuckoo hashing and multiqueues for efficient key-value management.

Provides theoretical analysis within the standard two-level memory model.

Abstract

Many data structures support dictionaries, also known as maps or associative arrays, which store and manage a set of key-value pairs. A \emph{multimap} is generalization that allows multiple values to be associated with the same key. For example, the inverted file data structure that is used prevalently in the infrastructure supporting search engines is a type of multimap, where words are used as keys and document pointers are used as values. We study the multimap abstract data type and how it can be implemented efficiently online in external memory frameworks, with constant expected I/O performance. The key technique used to achieve our results is a combination of cuckoo hashing using buckets that hold multiple items with a multiqueue implementation to cope with varying numbers of values per key. Our external-memory results are for the standard two-level memory model.