Generative Retrieval Overcomes Limitations of Dense Retrieval but Struggles with Identifier Ambiguity

TL;DR

This paper evaluates generative retrieval models, demonstrating their strengths on synthetic datasets and identifying key challenges like identifier ambiguity that limit their effectiveness.

Contribution

It provides the first evaluation of generative retrieval on the LIMIT dataset, highlighting both its advantages and the decoding issues affecting performance.

Findings

Generative retrieval outperforms dense and sparse methods on the LIMIT dataset.

Adding hard negatives significantly degrades all models' performance.

Identifier ambiguity causes decoding failures in generative retrieval.

Abstract

While dense retrieval models, which embed queries and documents into a shared low-dimensional space, have gained widespread popularity, they were shown to exhibit important theoretical limitations and considerably lag behind traditional sparse retrieval models in certain settings. Generative retrieval has emerged as an alternative approach to dense retrieval by using a language model to predict query-document relevance directly. In this paper, we demonstrate strengths and weaknesses of generative retrieval approaches using a simple synthetic dataset, called LIMIT, that was previously introduced to empirically demonstrate the theoretical limitations of embedding-based retrieval but was not used to evaluate generative retrieval. We close this research gap and show that generative retrieval achieves the best performance on this dataset without any additional training required (0.92 and 0.99 R@2 for SEAL and MINDER, respectively), compared to dense approaches (< 0.03 Recall@2) and BM25 (0.86 R@2). However, we then proceed to extend the original LIMIT dataset by adding simple hard negative samples and observe the performance degrading for all the models including the generative retrieval models (0.51 R@2) as well as BM25 (0.21 R@2). Error analysis identifies a failure in the decoding mechanism, caused by the inability to produce identifiers that are unique to relevant documents. Future generative retrieval must address these issues, either by designing identifiers that are more suitable to the decoding process or by adapting decoding and scoring algorithms to preserve relevance signals.