Exploring Domain Shift in Extractive Text Summarization

TL;DR

This paper investigates how domain shift affects extractive text summarization, extending the domain concept, re-purposing a dataset, and evaluating strategies to improve model generalization across different data sources.

Contribution

It introduces a new domain definition for summarization, a multi-domain dataset, and evaluates strategies to mitigate domain shift in neural summarization models.

Findings

Different strategies show varied effectiveness in handling domain shift.

Re-purposed dataset enables better evaluation of domain generalization.

Meta-learning methods improve cross-domain summarization performance.

Abstract

Although domain shift has been well explored in many NLP applications, it still has received little attention in the domain of extractive text summarization. As a result, the model is under-utilizing the nature of the training data due to ignoring the difference in the distribution of training sets and shows poor generalization on the unseen domain. With the above limitation in mind, in this paper, we first extend the conventional definition of the domain from categories into data sources for the text summarization task. Then we re-purpose a multi-domain summarization dataset and verify how the gap between different domains influences the performance of neural summarization models. Furthermore, we investigate four learning strategies and examine their abilities to deal with the domain shift problem. Experimental results on three different settings show their different characteristics in our new testbed. Our source code including \textit{BERT-based}, \textit{meta-learning} methods for multi-domain summarization learning and the re-purposed dataset \textsc{Multi-SUM} will be available on our project: \url{http://pfliu.com/TransferSum/}.