Rethinking On-policy Optimization for Query Augmentation

TL;DR

This paper systematically compares prompting-based and RL-based query augmentation methods for IR, revealing that simple prompting often matches or exceeds RL performance, and introduces a hybrid approach, OPQE, that outperforms both.

Contribution

It provides the first consistent comparison of query augmentation techniques and proposes a novel hybrid method, OPQE, combining prompting and RL for improved retrieval performance.

Findings

Prompting-based augmentation often matches or surpasses RL-based methods.

OPQE outperforms standalone prompting and RL approaches.

Hybrid approach effectively merges flexibility of prompting with targeted optimization.

Abstract

Recent advances in large language models (LLMs) have led to a surge of interest in query augmentation for information retrieval (IR). Two main approaches have emerged. The first prompts LLMs to generate answers or pseudo-documents that serve as new queries, relying purely on the model's parametric knowledge or contextual information. The second applies reinforcement learning (RL) to fine-tune LLMs for query rewriting, directly optimizing retrieval metrics. While having respective advantages and limitations, the two approaches have not been compared under consistent experimental conditions. In this work, we present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks, including evidence-seeking, ad hoc, and tool retrieval. Our key finding is that simple, training-free query augmentation often performs on par with, or even surpasses, more expensive RL-based counterparts, especially when using powerful LLMs. Motivated by this discovery, we introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), which, instead of rewriting a query, the LLM policy learns to generate a pseudo-document that maximizes retrieval performance, thus merging the flexibility and generative structure of prompting with the targeted optimization of RL. We show OPQE outperforms both standalone prompting and RL-based rewriting, demonstrating that a synergistic approach yields the best results. Our implementation is made available to facilitate reproducibility.