The State and Fate of Summarization Datasets: A Survey

TL;DR

This survey analyzes 133 summarization datasets across many languages, highlighting issues like lack of resources for low-resource languages and over-reliance on news data, while providing tools to improve dataset understanding and research coherence.

Contribution

It introduces a comprehensive ontology for summarization datasets, analyzes dataset properties, and offers tools to facilitate better dataset discovery and research standardization.

Findings

Limited high-quality datasets for low-resource languages

Over-reliance on news and distant supervision datasets

Provided a web interface and data card template for dataset exploration

Abstract

Automatic summarization has consistently attracted attention due to its versatility and wide application in various downstream tasks. Despite its popularity, we find that annotation efforts have largely been disjointed, and have lacked common terminology. Consequently, it is challenging to discover existing resources or identify coherent research directions. To address this, we survey a large body of work spanning 133 datasets in over 100 languages, creating a novel ontology covering sample properties, collection methods and distribution. With this ontology we make key observations, including the lack in accessible high-quality datasets for low-resource languages, and the field's over-reliance on the news domain and on automatically collected distant supervision. Finally, we make available a web interface that allows users to interact and explore our ontology and dataset collection, as well as a template for a summarization data card, which can be used to streamline future research into a more coherent body of work.