Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries

TL;DR

This paper explores using large language models with chain-of-thought prompting for multi-round post-editing to improve the faithfulness of news summaries, showing it can outperform prior methods and enhance correction success.

Contribution

It introduces a multi-round, chain-of-thought prompting approach for LLM-based post-editing of summaries, demonstrating improved faithfulness correction over single-round methods.

Findings

Chain-of-thought prompting correlates well with human judgments of faithfulness.

Multi-round editing further improves summary accuracy.

Prompting with error-type reasoning is effective for factual correction.

Abstract

Recent large language models (LLMs) have demonstrated a remarkable ability to perform natural language understanding and generation tasks. In this work, we investigate the use of LLMs for evaluating faithfulness in news summarization, finding that it achieves a strong correlation with human judgments. We further investigate LLMs' capabilities as a faithfulness post-editor, experimenting with different chain-of-thought prompts for locating and correcting factual inconsistencies between a generated summary and the source news document and are able to achieve a higher editing success rate than was reported in prior work. We perform both automated and human evaluations of the post-edited summaries, finding that prompting LLMs using chain-of-thought reasoning about factual error types is an effective faithfulness post-editing strategy, performing comparably to fine-tuned post-editing models. We also demonstrate that multiple rounds of post-editing, which has not previously been explored, can be used to gradually improve the faithfulness of summaries whose errors cannot be fully corrected in a single round.