Self-Supervised Learning for Contextualized Extractive Summarization

TL;DR

This paper introduces three self-supervised auxiliary tasks for extractive summarization that enhance document-level context understanding, leading to improved performance over existing models on the CNN/DM dataset.

Contribution

The paper proposes novel self-supervised auxiliary pre-training tasks that explicitly capture document-level context for extractive summarization.

Findings

Pre-training with auxiliary tasks improves summarization performance.

The approach outperforms previous state-of-the-art models.

Simple models after pre-training achieve superior results.

Abstract

Existing models for extractive summarization are usually trained from scratch with a cross-entropy loss, which does not explicitly capture the global context at the document level. In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion. Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks. Furthermore, we show that after pre-training, a clean model with simple building blocks is able to outperform previous state-of-the-art that are carefully designed.