On the Robustness of Generative Retrieval Models: An Out-of-Distribution Perspective

TL;DR

This paper investigates the out-of-distribution robustness of generative retrieval models in information retrieval, highlighting their vulnerabilities and the need for improvement in handling new query types and tasks.

Contribution

It introduces a taxonomy for OOD robustness in retrieval, and empirically compares generative and dense retrieval models' generalization capabilities.

Findings

Generative retrieval models show limited OOD robustness.

Empirical analysis reveals vulnerabilities to new query types.

Enhancement of robustness is necessary for real-world applications.

Abstract

Recently, we have witnessed generative retrieval increasingly gaining attention in the information retrieval (IR) field, which retrieves documents by directly generating their identifiers. So far, much effort has been devoted to developing effective generative retrieval models. There has been less attention paid to the robustness perspective. When a new retrieval paradigm enters into the real-world application, it is also critical to measure the out-of-distribution (OOD) generalization, i.e., how would generative retrieval models generalize to new distributions. To answer this question, firstly, we define OOD robustness from three perspectives in retrieval problems: 1) The query variations; 2) The unforeseen query types; and 3) The unforeseen tasks. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of several representative generative retrieval models against dense retrieval models. The empirical results indicate that the OOD robustness of generative retrieval models requires enhancement. We hope studying the OOD robustness of generative retrieval models would be advantageous to the IR community.