Unleashing Potential of Evidence in Knowledge-Intensive Dialogue Generation

TL;DR

This paper presents a novel framework that leverages large language models to automatically generate and utilize evidence labels, significantly enhancing the factual accuracy and coherence of knowledge-intensive dialogue generation.

Contribution

It introduces an automatic evidence generation framework and an evidence-focused attention mechanism to better incorporate relevant evidence into dialogue models.

Findings

Improved coherence and factual consistency (+3~+5 points)

Effective evidence label augmentation enhances performance

Refined attention mechanisms focus on relevant evidence segments

Abstract

Incorporating external knowledge into dialogue generation (KIDG) is crucial for improving the correctness of response, where evidence fragments serve as knowledgeable snippets supporting the factual dialogue replies. However, introducing irrelevant content often adversely impacts reply quality and easily leads to hallucinated responses. Prior work on evidence retrieval and integration in dialogue systems falls short of fully leveraging existing evidence since the model fails to locate useful fragments accurately and overlooks hidden evidence labels within the KIDG dataset. To fully Unleash the potential of evidence, we propose a framework to effectively incorporate Evidence in knowledge-Intensive Dialogue Generation (u-EIDG). Specifically, we introduce an automatic evidence generation framework that harnesses the power of Large Language Models (LLMs) to mine reliable evidence veracity labels from unlabeled data. By utilizing these evidence labels, we train a reliable evidence indicator to effectively identify relevant evidence from retrieved passages. Furthermore, we propose an evidence-augmented generator with an evidence-focused attention mechanism, which allows the model to concentrate on evidenced segments. Experimental results on MultiDoc2Dial demonstrate the efficacy of evidential label augmentation and refined attention mechanisms in improving model performance. Further analysis confirms that the proposed method outperforms other baselines (+3~+5 points) regarding coherence and factual consistency.