DRS: Deep Question Reformulation With Structured Output

TL;DR

The paper introduces DRS, a zero-shot structured output method that significantly improves question reformulation accuracy in large language models, enabling better extraction of relevant information from unfamiliar texts.

Contribution

DRS combines LLMs with a DFS-based algorithm to enhance question reformulation capabilities without requiring training data.

Findings

DRS improves GPT-3.5 reformulation accuracy from 23.03% to 70.42%.

DRS enhances open-source models' reformulation accuracy from 26.35% to 56.75%.

Structured reformulation significantly aids in extracting relevant information from new documents.

Abstract

Question answering represents a core capability of large language models (LLMs). However, when individuals encounter unfamiliar knowledge in texts, they often formulate questions that the text itself cannot answer due to insufficient understanding of the underlying information. Recent studies reveal that while LLMs can detect unanswerable questions, they struggle to assist users in reformulating these questions. Even advanced models like GPT-3.5 demonstrate limited effectiveness in this regard. To address this limitation, we propose DRS: Deep Question Reformulation with Structured Output, a novel zero-shot method aimed at enhancing LLMs ability to assist users in reformulating questions to extract relevant information from new documents. DRS combines the strengths of LLMs with a DFS-based algorithm to iteratively explore potential entity combinations and constrain outputs using predefined entities. This structured approach significantly enhances the reformulation capabilities of LLMs. Comprehensive experimental evaluations demonstrate that DRS improves the reformulation accuracy of GPT-3.5 from $23.03\%$ to $70.42\%$, while also enhancing the performance of open-source models, such as Gemma2-9B, from $26.35\%$ to $56.75\%$.