Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval

TL;DR

This paper establishes optimal bounds for incremental and dynamic retrieval data structures with polynomial universes, revealing a phase transition where incremental space redundancy decreases as value size approaches log n.

Contribution

It completes the theoretical understanding of incremental and dynamic retrieval structures and uncovers a surprising phase transition in their space complexity.

Findings

Optimal bounds for incremental and dynamic retrieval structures.

Discovery of a phase transition in space redundancy as value size approaches log n.

Incremental setting's space redundancy can be reduced to linear, contrary to previous beliefs.

Abstract

Retrieval data structures are data structures that answer key-value queries without paying the space overhead of explicitly storing keys. The problem can be formulated in four settings (static, value-dynamic, incremental, or dynamic), each of which offers different levels of dynamism to the user. In this paper, we establish optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe. Our results complete a line of work that has spanned more than two decades, and also come with a surprise: the incremental setting, which has long been viewed as essentially equivalent to the dynamic one, actually has a phase transition, in which, as the value size $v$ approaches $\log n$, the optimal space redundancy actually begins to shrink, going from roughly $n \log \log n$ (which has long been thought to be optimal) all the way down to $\Theta(n)$ (which is the optimal bound even for the seemingly much-easier value-dynamic setting).