EmailSum: Abstractive Email Thread Summarization

TL;DR

This paper introduces EmailSum, a new dataset for email thread summarization, and conducts extensive experiments revealing challenges in current models and the need for better evaluation metrics, emphasizing human judgment.

Contribution

The paper presents a large, human-annotated dataset for email thread summarization and provides a comprehensive empirical study of various summarization techniques and evaluation methods.

Findings

Current models struggle with understanding sender intent and roles.

Automatic metrics like ROUGE and BERTScore are weakly correlated with human judgments.

Human evaluation is crucial for assessing summary quality.

Abstract

Recent years have brought about an interest in the challenging task of summarizing conversation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch up with the decisions made and thus improve our work or communication efficiency. To spur research in thread summarization, we have developed an abstractive Email Thread Summarization (EmailSum) dataset, which contains human-annotated short (<30 words) and long (<100 words) summaries of 2549 email threads (each containing 3 to 10 emails) over a wide variety of topics. We perform a comprehensive empirical study to explore different summarization techniques (including extractive and abstractive methods, single-document and hierarchical models, as well as transfer and semisupervised learning) and conduct human evaluations on both short and long summary generation tasks. Our results reveal the key challenges of current abstractive summarization models in this task, such as understanding the sender's intent and identifying the roles of sender and receiver. Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task. Hence, we emphasize the importance of human evaluation and the development of better metrics by the community. Our code and summary data have been made available at: https://github.com/ZhangShiyue/EmailSum