A Step Closer to Comprehensive Answers: Constrained Multi-Stage Question Decomposition with Large Language Models

TL;DR

This paper introduces the Decompose-and-Query framework, enabling large language models to better handle complex multi-hop questions by guiding reasoning and restricting reliance on external knowledge, thereby reducing hallucinations.

Contribution

The paper presents a novel framework that improves multi-hop question answering by combining guided decomposition with external knowledge utilization and hallucination mitigation.

Findings

D&Q matches ChatGPT in 67% of cases on ChitChatQA.

Achieved an F1 score of 59.6% on HotPotQA question-only setting.

Effectively reduces hallucinations in complex question answering.

Abstract

While large language models exhibit remarkable performance in the Question Answering task, they are susceptible to hallucinations. Challenges arise when these models grapple with understanding multi-hop relations in complex questions or lack the necessary knowledge for a comprehensive response. To address this issue, we introduce the "Decompose-and-Query" framework (D&Q). This framework guides the model to think and utilize external knowledge similar to ReAct, while also restricting its thinking to reliable information, effectively mitigating the risk of hallucinations. Experiments confirm the effectiveness of D&Q: On our ChitChatQA dataset, D&Q does not lose to ChatGPT in 67% of cases; on the HotPotQA question-only setting, D&Q achieved an F1 score of 59.6%. Our code is available at https://github.com/alkaidpku/DQ-ToolQA.