Parallel Hierarchical Transformer with Attention Alignment for Abstractive Multi-Document Summarization

TL;DR

This paper introduces a Parallel Hierarchical Transformer with attention alignment for multi-document summarization, improving coverage and quality of generated summaries by leveraging hierarchical attention and attention calibration.

Contribution

The study proposes a novel hierarchical Transformer architecture with attention alignment for MDS, enhancing dependency modeling and summary coverage over existing models.

Findings

Improved ROUGE scores over baselines.

Higher quality summaries in human evaluations.

Efficient processing with low computational cost.

Abstract

In comparison to single-document summarization, abstractive Multi-Document Summarization (MDS) brings challenges on the representation and coverage of its lengthy and linked sources. This study develops a Parallel Hierarchical Transformer (PHT) with attention alignment for MDS. By incorporating word- and paragraph-level multi-head attentions, the hierarchical architecture of PHT allows better processing of dependencies at both token and document levels. To guide the decoding towards a better coverage of the source documents, the attention-alignment mechanism is then introduced to calibrate beam search with predicted optimal attention distributions. Based on the WikiSum data, a comprehensive evaluation is conducted to test improvements on MDS by the proposed architecture. By better handling the inner- and cross-document information, results in both ROUGE and human evaluation suggest that our hierarchical model generates summaries of higher quality relative to other Transformer-based baselines at relatively low computational cost.