Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation

TL;DR

This study investigates how prompt rewriting by LLMs can improve response quality in human-AI conversations, showing that rephrasing ineffective prompts enhances responses while maintaining user intent, especially in longer dialogues.

Contribution

It is the first LLM-centric analysis of real human-AI chat data focusing on prompt rewriting to improve response relevance and quality.

Findings

Rephrasing prompts improves response quality.

Longer conversations benefit more from prompt rewriting.

LLMs make plausible assumptions about user intent.

Abstract

Human-LLM conversations are increasingly becoming more pervasive in peoples' professional and personal lives, yet many users still struggle to elicit helpful responses from LLM Chatbots. One of the reasons for this issue is users' lack of understanding in crafting effective prompts that accurately convey their information needs. Meanwhile, the existence of real-world conversational datasets on the one hand, and the text understanding faculties of LLMs on the other, present a unique opportunity to study this problem, and its potential solutions at scale. Thus, in this paper we present the first LLM-centric study of real human-AI chatbot conversations, focused on investigating aspects in which user queries fall short of expressing information needs, and the potential of using LLMs to rewrite suboptimal user prompts. Our findings demonstrate that rephrasing ineffective prompts can elicit better responses from a conversational system, while preserving the user's original intent. Notably, the performance of rewrites improves in longer conversations, where contextual inferences about user needs can be made more accurately. Additionally, we observe that LLMs often need to -- and inherently do -- make \emph{plausible} assumptions about a user's intentions and goals when interpreting prompts. Our findings largely hold true across conversational domains, user intents, and LLMs of varying sizes and families, indicating the promise of using prompt rewriting as a solution for better human-AI interactions.