Analyzing and Evaluating Faithfulness in Dialogue Summarization
Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, Haizhou Li

TL;DR
This paper investigates the faithfulness of dialogue summaries, revealing that over 35% contain factual inconsistencies, and introduces a new model-level evaluation method to improve assessment of factual correctness.
Contribution
It provides a systematic human analysis of faithfulness in dialogue summarization and proposes a novel evaluation approach using multi-choice questions for better factual assessment.
Findings
35% of summaries are factually inconsistent
Proposed evaluation method correlates well with human judgment
Released dataset and toolkit for future research
Abstract
Dialogue summarization is abstractive in nature, making it suffer from factual errors. The factual correctness of summaries has the highest priority before practical applications. Many efforts have been made to improve faithfulness in text summarization. However, there is a lack of systematic study on dialogue summarization systems. In this work, we first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. Furthermore, we present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations. Experimental results show that our evaluation schema is a strong proxy for the factual correctness of summarization models. The human-annotated faithfulness samples and the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
