Hierarchical Transformers for Multi-Document Summarization

TL;DR

This paper introduces a hierarchical Transformer model for multi-document summarization that captures cross-document relationships using attention mechanisms and graph representations, leading to improved summarization performance.

Contribution

The paper presents a novel hierarchical Transformer architecture that effectively models relationships among multiple documents for summarization, outperforming existing baselines.

Findings

Significant improvements on the WikiSum dataset.

Effective encoding of cross-document relationships.

Utilization of explicit graph representations enhances performance.

Abstract

In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial improvements over several strong baselines.