Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter

TL;DR

This paper introduces a question rewriter that improves large language models' answers by clarifying user questions through prompt optimization, trained without human annotations, and validated across multiple datasets and models.

Contribution

It presents a novel, cost-effective method for question clarification in LLMs using preference optimization, enhancing answer quality in LFQA tasks.

Findings

Improved answer quality across multiple LLMs

Effective training without human annotations

Framework applicable to prompt optimization

Abstract

Large Language Models (LLMs) have demonstrated significant capabilities, particularly in the domain of question answering (QA). However, their effectiveness in QA is often undermined by the vagueness of user questions. To address this issue, we introduce single-round instance-level prompt optimization, referred to as question rewriter. By enhancing the intelligibility of human questions for black-box LLMs, our question rewriter improves the quality of generated answers. The rewriter is optimized using direct preference optimization based on feedback collected from automatic criteria for evaluating generated answers; therefore, its training does not require costly human annotations. The experiments across multiple black-box LLMs and long-form question answering (LFQA) datasets demonstrate the efficacy of our method. This paper provides a practical framework for training question rewriters and sets a precedent for future explorations in prompt optimization within LFQA tasks. Code is available at https://github.com/3244we/Question-Rewriter.