The Power of Summary-Source Alignments

TL;DR

This paper enhances multi-document summarization by applying detailed, manual summary-source alignments at the proposition level, creating datasets for multiple tasks, and providing baselines to advance research in the field.

Contribution

It introduces a fine-grained, multi-document alignment framework, manual annotation of alignments, and the creation of multiple datasets for diverse summarization tasks.

Findings

Manual alignment improves dataset quality for summarization tasks.

Multiple datasets enable benchmarking across six different tasks.

Baseline models demonstrate the utility of the proposed datasets.

Abstract

Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection, followed by text generation. In this context, alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data for some of the component tasks. Yet, this enabling alignment step has usually been applied heuristically on the sentence level on a limited number of subtasks. In this paper, we propose extending the summary-source alignment framework by (1) applying it at the more fine-grained proposition span level, (2) annotating alignment manually in a multi-document setup, and (3) revealing the great potential of summary-source alignments to yield several datasets for at least six different tasks. Specifically, for each of the tasks, we release a manually annotated test set that was derived automatically from the alignment annotation. We also release development and train sets in the same way, but from automatically derived alignments. Using the datasets, each task is demonstrated with baseline models and corresponding evaluation metrics to spur future research on this broad challenge.