Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization

TL;DR

This paper analyzes attention weights in graph-based transformer models for multi-document summarization to enhance their explainability, revealing correlations between attention, source origin, and positional bias.

Contribution

It introduces a method to interpret attention weights in GraphSum, improving understanding of source contributions and positional influences in multi-document summarization.

Findings

Paragraph-level representations outperform sentence-level in summarization.

High correlation between attention weights and source similarity metrics.

Attention patterns reveal positional biases in summaries.

Abstract

Modern multi-document summarization (MDS) methods are based on transformer architectures. They generate state of the art summaries, but lack explainability. We focus on graph-based transformer models for MDS as they gained recent popularity. We aim to improve the explainability of the graph-based MDS by analyzing their attention weights. In a graph-based MDS such as GraphSum, vertices represent the textual units, while the edges form some similarity graph over the units. We compare GraphSum's performance utilizing different textual units, i. e., sentences versus paragraphs, on two news benchmark datasets, namely WikiSum and MultiNews. Our experiments show that paragraph-level representations provide the best summarization performance. Thus, we subsequently focus oAnalysisn analyzing the paragraph-level attention weights of GraphSum's multi-heads and decoding layers in order to improve the explainability of a transformer-based MDS model. As a reference metric, we calculate the ROUGE scores between the input paragraphs and each sentence in the generated summary, which indicate source origin information via text similarity. We observe a high correlation between the attention weights and this reference metric, especially on the the later decoding layers of the transformer architecture. Finally, we investigate if the generated summaries follow a pattern of positional bias by extracting which paragraph provided the most information for each generated summary. Our results show that there is a high correlation between the position in the summary and the source origin.