A Systematic Study of Pseudo-Relevance Feedback with LLMs

TL;DR

This paper systematically investigates how different feedback sources and models in pseudo-relevance feedback with large language models affect retrieval effectiveness, revealing key factors for optimal design.

Contribution

It provides a controlled experimental analysis of PRF design dimensions, clarifying the impact of feedback source and model choices on effectiveness.

Findings

Feedback model choice significantly influences PRF success.

LLM-generated feedback is most cost-effective.

Corpus-derived feedback is best with strong first-stage retrievers.

Abstract

Pseudo-relevance feedback (PRF) methods built on large language models (LLMs) can be organized along two key design dimensions: the feedback source, which is where the feedback text is derived from and the feedback model, which is how the given feedback text is used to refine the query representation. However, the independent role that each dimension plays is unclear, as both are often entangled in empirical evaluations. In this paper, we address this gap by systematically studying how the choice of feedback source and feedback model impact PRF effectiveness through controlled experimentation. Across 13 low-resource BEIR tasks with five LLM PRF methods, our results show: (1) the choice of feedback model can play a critical role in PRF effectiveness; (2) feedback derived solely from LLM-generated text provides the most cost-effective solution; and (3) feedback derived from the corpus is most beneficial when utilizing candidate documents from a strong first-stage retriever. Together, our findings provide a better understanding of which elements in the PRF design space are most important.