Improving Abstractive Text Summarization with History Aggregation

TL;DR

This paper introduces a history aggregation mechanism for Transformer-based models to improve long text understanding in abstractive summarization, resulting in higher quality summaries on CNN/DailyMail.

Contribution

It proposes a novel aggregation mechanism that enhances Transformer encoders' ability to handle long texts in summarization tasks.

Findings

Achieves higher ROUGE scores than baseline models.

Improves encoder memory capacity for long input texts.

Demonstrates effectiveness on CNN/DailyMail dataset.

Abstract

Recent neural sequence to sequence models have provided feasible solutions for abstractive summarization. However, such models are still hard to tackle long text dependency in the summarization task. A high-quality summarization system usually depends on strong encoder which can refine important information from long input texts so that the decoder can generate salient summaries from the encoder's memory. In this paper, we propose an aggregation mechanism based on the Transformer model to address the challenge of long text representation. Our model can review history information to make encoder hold more memory capacity. Empirically, we apply our aggregation mechanism to the Transformer model and experiment on CNN/DailyMail dataset to achieve higher quality summaries compared to several strong baseline models on the ROUGE metrics.