An Exploration of Post-Editing Effectiveness in Text Summarization

TL;DR

This study investigates the effectiveness of human post-editing of AI-generated summaries, revealing that it can improve quality in some cases but not when summaries contain inaccuracies, with implications for future systems.

Contribution

It provides empirical insights into when human post-editing enhances AI-generated summaries and explores strategies and needs for better human-AI collaboration in summarization.

Findings

Post-editing improves summary quality when domain knowledge is limited.

Inaccurate AI summaries are less effectively improved through post-editing.

Participants use diverse editing strategies and require different levels of assistance.

Abstract

Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of "post-editing" AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants' different editing strategies and needs for assistance offer implications for future human-AI summarization systems.