Transformer Memory as a Differentiable Search Index
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh, Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W., Cohen, Donald Metzler

TL;DR
This paper introduces the Differentiable Search Index (DSI), a novel approach where a Transformer model encodes an entire corpus within its parameters to perform direct text-to-docid retrieval, simplifying and enhancing information retrieval.
Contribution
The paper presents DSI, a new paradigm that enables direct retrieval by training a Transformer to map queries to document IDs, outperforming traditional baselines and demonstrating strong generalization.
Findings
DSI outperforms dual encoder baselines.
DSI surpasses BM25 in zero-shot retrieval.
Model and corpus size variations affect performance.
Abstract
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text and Document Classification Technologies
MethodsAttention Is All You Need · Linear Layer · Softmax · Layer Normalization · Multi-Head Attention · Byte Pair Encoding · Dense Connections · Absolute Position Encodings · Dropout · Label Smoothing
