HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information

TL;DR

This paper introduces HiStruct+, a novel extractive summarization model that explicitly incorporates hierarchical structure information from texts, leading to significant improvements over state-of-the-art methods on multiple datasets.

Contribution

The paper proposes a new method to encode and inject hierarchical structure information into a Transformer-based summarization model, enhancing its performance.

Findings

HiStruct+ outperforms strong baselines on PubMed and arXiv datasets.

Hierarchical structure information significantly boosts summarization quality.

The model's improvements are more pronounced with datasets having clearer hierarchical structures.

Abstract

Transformer-based language models usually treat texts as linear sequences. However, most texts also have an inherent hierarchical structure, i.e., parts of a text can be identified using their position in this hierarchy. In addition, section titles usually indicate the common topic of their respective sentences. We propose a novel approach to formulate, extract, encode and inject hierarchical structure information explicitly into an extractive summarization model based on a pre-trained, encoder-only Transformer language model (HiStruct+ model), which improves SOTA ROUGEs for extractive summarization on PubMed and arXiv substantially. Using various experimental settings on three datasets (i.e., CNN/DailyMail, PubMed and arXiv), our HiStruct+ model outperforms a strong baseline collectively, which differs from our model only in that the hierarchical structure information is not injected. It is also observed that the more conspicuous hierarchical structure the dataset has, the larger improvements our method gains. The ablation study demonstrates that the hierarchical position information is the main contributor to our model's SOTA performance.