Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMs

TL;DR

This paper presents a prompt-guided in-context learning approach using large language models for conversational query rewriting, achieving superior results in low-resource settings without explicit fine-tuning.

Contribution

It introduces a novel prompt-based method that leverages LLMs for few-shot conversational query rewriting, reducing the need for labeled data and complex training.

Findings

Outperforms supervised and contrastive co-training baselines on TREC and Taskmaster-1 datasets.

Ablation studies show the importance of in-context examples.

Human evaluations confirm improved fluency, relevance, and context understanding.

Abstract

Conversational query rewriting is crucial for effective conversational search, yet traditional supervised methods require substantial labeled data, which is scarce in low-resource settings. This paper introduces Prompt-Guided In-Context Learning, a novel approach that leverages the in-context learning capabilities of Large Language Models (LLMs) for few-shot conversational query rewriting. Our method employs carefully designed prompts, incorporating task descriptions, input/output format specifications, and a small set of illustrative examples, to guide pre-trained LLMs to generate context-independent queries without explicit fine-tuning. Extensive experiments on benchmark datasets, TREC and Taskmaster-1, demonstrate that our approach significantly outperforms strong baselines, including supervised models and contrastive co-training methods, across various evaluation metrics such as BLEU, ROUGE-L, Success Rate, and MRR. Ablation studies confirm the importance of in-context examples, and human evaluations further validate the superior fluency, relevance, and context utilization of our generated rewrites. The results highlight the potential of prompt-guided in-context learning as an efficient and effective paradigm for low-resource conversational query rewriting, reducing the reliance on extensive labeled data and complex training procedures.