LLM-Assisted Pseudo-Relevance Feedback

TL;DR

This paper introduces a hybrid pseudo-relevance feedback method that uses an LLM-based filtering step to improve query expansion in information retrieval, reducing noise and enhancing relevance.

Contribution

It proposes a novel approach combining classical PRF with LLM filtering to enhance robustness and interpretability in query expansion.

Findings

Improved retrieval performance over baseline methods.

Effective reduction of noisy or tangential content in top-ranked documents.

Demonstrated across multiple datasets and metrics.

Abstract

Query expansion is a long-standing technique to mitigate vocabulary mismatch in ad hoc Information Retrieval. Pseudo-relevance feedback methods, such as RM3, estimate an expanded query model from the top-ranked documents, but remain vulnerable to topic drift when early results include noisy or tangential content. Recent approaches instead prompt Large Language Models to generate synthetic expansions or query variants. While effective, these methods risk hallucinations and misalignment with collection-specific terminology. We propose a hybrid alternative that preserves the robustness and interpretability of classical PRF while leveraging LLM semantic judgement. Our method inserts an LLM-based filtering stage prior to RM3 estimation: the LLM judges the documents in the initial top-$k$ ranking, and RM3 is computed only over those accepted as relevant. This simple intervention improves over blind PRF and a strong baseline across several datasets and metrics.