On the Robustness of Generative Information Retrieval Models

TL;DR

This paper evaluates the out-of-distribution robustness of generative information retrieval models across various scenarios, revealing their current limitations and emphasizing the need for more reliable IR systems.

Contribution

It provides the first comprehensive empirical analysis of the OOD robustness of generative IR models across multiple challenging scenarios.

Findings

Generative IR models show limited robustness to OOD scenarios.

Compared to dense retrieval models, generative models are less reliable in new distributions.

The study highlights areas for improving the robustness of generative IR methods.

Abstract

Generative information retrieval methods retrieve documents by directly generating their identifiers. Much effort has been devoted to developing effective generative IR models. Less attention has been paid to the robustness of these models. It is critical to assess the out-of-distribution (OOD) generalization of generative IR models, i.e., how would such models generalize to new distributions? To answer this question, we focus on OOD scenarios from four perspectives in retrieval problems: (i)query variations; (ii)unseen query types; (iii)unseen tasks; and (iv)corpus expansion. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of representative generative IR models against dense retrieval models. Our empirical results indicate that the OOD robustness of generative IR models is in need of improvement. By inspecting the OOD robustness of generative IR models we aim to contribute to the development of more reliable IR models. The code is available at \url{https://github.com/Davion-Liu/GR_OOD}.