Towards Understanding Omission in Dialogue Summarization

TL;DR

This paper introduces the OLDS dataset with omission labels for dialogue summarization, highlighting the importance of detecting omitted information to improve summarization quality and providing tools for further research.

Contribution

The paper presents a new dataset with omission labels and demonstrates the significance of omission detection in enhancing dialogue summarization.

Findings

Providing ground-truth omission labels improves summarization quality.

The OLDS dataset supports training and evaluation of omission detection.

Omission detection is crucial for reducing information loss in dialogue summaries.

Abstract

Dialogue summarization aims to condense the lengthy dialogue into a concise summary, and has recently achieved significant progress. However, the result of existing methods is still far from satisfactory. Previous works indicated that omission is a major factor in affecting the quality of summarization, but few of them have further explored the omission problem, such as how omission affects summarization results and how to detect omission, which is critical for reducing omission and improving summarization quality. Moreover, analyzing and detecting omission relies on summarization datasets with omission labels (i.e., which dialogue utterances are omitted in the summarization), which are not available in the current literature. In this paper, we propose the OLDS dataset, which provides high-quality Omission Labels for Dialogue Summarization. By analyzing this dataset, we find that a large improvement in summarization quality can be achieved by providing ground-truth omission labels for the summarization model to recover omission information, which demonstrates the importance of omission detection for omission mitigation in dialogue summarization. Therefore, we formulate an omission detection task and demonstrate our proposed dataset can support the training and evaluation of this task well. We also call for research action on omission detection based on our proposed datasets. Our dataset and codes are publicly available.