Text Summarization with Pretrained Encoders

TL;DR

This paper demonstrates how BERT can be effectively adapted for both extractive and abstractive text summarization, introducing novel models and training strategies that achieve state-of-the-art results.

Contribution

The paper presents a unified BERT-based framework for extractive and abstractive summarization, including a new document encoder and fine-tuning methods to improve summary quality.

Findings

Achieves state-of-the-art results on three datasets

Introduces a novel document-level BERT encoder

Develops a two-staged fine-tuning approach

Abstract

Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings. Our code is available at https://github.com/nlpyang/PreSumm