How does Feedback Signal Quality Impact Effectiveness of Pseudo Relevance Feedback for Passage Retrieval?

TL;DR

This paper investigates how the quality of feedback signals influences the effectiveness of various pseudo-relevance feedback methods in passage retrieval, revealing that different methods respond differently to feedback quality.

Contribution

It systematically analyzes the impact of feedback signal quality on multiple PRF methods, highlighting their varying robustness and guiding future application choices.

Findings

Feedback signal quality significantly affects PRF effectiveness.

Different PRF methods respond differently to feedback quality.

Understanding this can improve retrieval performance and method selection.

Abstract

Pseudo-Relevance Feedback (PRF) assumes that the top results retrieved by a first-stage ranker are relevant to the original query and uses them to improve the query representation for a second round of retrieval. This assumption however is often not correct: some or even all of the feedback documents may be irrelevant. Indeed, the effectiveness of PRF methods may well depend on the quality of the feedback signal and thus on the effectiveness of the first-stage ranker. This aspect however has received little attention before. In this paper we control the quality of the feedback signal and measure its impact on a range of PRF methods, including traditional bag-of-words methods (Rocchio), and dense vector-based methods (learnt and not learnt). Our results show the important role the quality of the feedback signal plays on the effectiveness of PRF methods. Importantly, and surprisingly, our analysis reveals that not all PRF methods are the same when dealing with feedback signals of varying quality. These findings are critical to gain a better understanding of the PRF methods and of which and when they should be used, depending on the feedback signal quality, and set the basis for future research in this area.