Bottleneck-Minimal Indexing for Generative Document Retrieval

TL;DR

This paper introduces a novel information-theoretic approach to generative document retrieval, analyzing and minimizing the index's mutual information bottleneck to improve retrieval efficiency and effectiveness.

Contribution

It reformulates generative document retrieval using Shannon's rate-distortion theory and proposes a bottleneck-minimal indexing method that outperforms previous approaches.

Findings

The proposed method reduces the mutual information bottleneck in GDR.

Empirical results show improved retrieval performance on NQ320K and MARCO datasets.

The approach offers a theoretical framework for optimizing document indexes in GDR.

Abstract

We apply an information-theoretic perspective to reconsider generative document retrieval (GDR), in which a document $x \in X$ is indexed by $t \in T$, and a neural autoregressive model is trained to map queries $Q$ to $T$. GDR can be considered to involve information transmission from documents $X$ to queries $Q$, with the requirement to transmit more bits via the indexes $T$. By applying Shannon's rate-distortion theory, the optimality of indexing can be analyzed in terms of the mutual information, and the design of the indexes $T$ can then be regarded as a {\em bottleneck} in GDR. After reformulating GDR from this perspective, we empirically quantify the bottleneck underlying GDR. Finally, using the NQ320K and MARCO datasets, we evaluate our proposed bottleneck-minimal indexing method in comparison with various previous indexing methods, and we show that it outperforms those methods.