Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action

TL;DR

Conv-CoA introduces a dynamic reasoning and retrieval framework for open-domain conversational question answering, addressing hallucinations, reasoning, and retrieval challenges, and demonstrating superior accuracy and efficiency over 23 methods.

Contribution

The paper proposes a novel Conv-CoA framework with a reasoning-retrieval mechanism, including a Hopfield-based retriever and faith score, improving open-domain conversational QA performance.

Findings

Outperforms 23 state-of-the-art methods in accuracy and efficiency

Effectively reduces hallucinations and improves reasoning in conversations

Enhances retrieval accuracy with a resource-efficient Hopfield retriever

Abstract

We present a Conversational Chain-of-Action (Conv-CoA) framework for Open-domain Conversational Question Answering (OCQA). Compared with literature, Conv-CoA addresses three major challenges: (i) unfaithful hallucination that is inconsistent with real-time or domain facts, (ii) weak reasoning performance in conversational scenarios, and (iii) unsatisfying performance in conversational information retrieval. Our key contribution is a dynamic reasoning-retrieval mechanism that extracts the intent of the question and decomposes it into a reasoning chain to be solved via systematic prompting, pre-designed actions, updating the Contextual Knowledge Set (CKS), and a novel Hopfield-based retriever. Methodologically, we propose a resource-efficiency Hopfield retriever to enhance the efficiency and accuracy of conversational information retrieval within our actions. Additionally, we propose a conversational-multi-reference faith score (Conv-MRFS) to verify and resolve conflicts between retrieved knowledge and answers in conversations. Empirically, we conduct comparisons between our framework and 23 state-of-the-art methods across five different research directions and two public benchmarks. These comparisons demonstrate that our Conv-CoA outperforms other methods in both the accuracy and efficiency dimensions.