A Modern Perspective on Query Likelihood with Deep Generative Retrieval Models

TL;DR

This paper introduces a probabilistic deep generative retrieval paradigm that leverages neural models like Transformers, demonstrating improved passage retrieval performance and uncertainty estimation for relevance ranking.

Contribution

It formalizes a new deep generative retrieval framework, introduces a novel T-PGN model combining Transformers and Pointer Generator Networks, and shows its superior performance on passage retrieval tasks.

Findings

T-PGN outperforms other generative models in retrieval tasks.

Uncertainty estimation enhances query and collection understanding.

Generative models improve cut-off prediction accuracy.

Abstract

Existing neural ranking models follow the text matching paradigm, where document-to-query relevance is estimated through predicting the matching score. Drawing from the rich literature of classical generative retrieval models, we introduce and formalize the paradigm of deep generative retrieval models defined via the cumulative probabilities of generating query terms. This paradigm offers a grounded probabilistic view on relevance estimation while still enabling the use of modern neural architectures. In contrast to the matching paradigm, the probabilistic nature of generative rankers readily offers a fine-grained measure of uncertainty. We adopt several current neural generative models in our framework and introduce a novel generative ranker (T-PGN), which combines the encoding capacity of Transformers with the Pointer Generator Network model. We conduct an extensive set of evaluation experiments on passage retrieval, leveraging the MS MARCO Passage Re-ranking and TREC Deep Learning 2019 Passage Re-ranking collections. Our results show the significantly higher performance of the T-PGN model when compared with other generative models. Lastly, we demonstrate that exploiting the uncertainty information of deep generative rankers opens new perspectives to query/collection understanding, and significantly improves the cut-off prediction task.