Rethinking Transformer-based Multi-document Summarization: An Empirical Investigation

TL;DR

This paper empirically investigates Transformer-based multi-document summarization, analyzing model behaviors, training strategies, and issues like repetition, to guide future improvements in summary quality.

Contribution

It provides a comprehensive empirical analysis of Transformer models in MDS, highlighting the impact of document boundaries, model structures, and training strategies.

Findings

Document boundary separators significantly affect performance.

Decoder sensitivity to noise impacts summary quality.

Repetition correlates with high uncertainty scores.

Abstract

The utilization of Transformer-based models prospers the growth of multi-document summarization (MDS). Given the huge impact and widespread adoption of Transformer-based models in various natural language processing tasks, investigating their performance and behaviors in the context of MDS becomes crucial for advancing the field and enhancing the quality of summary. To thoroughly examine the behaviours of Transformer-based MDS models, this paper presents five empirical studies on (1) measuring the impact of document boundary separators quantitatively; (2) exploring the effectiveness of different mainstream Transformer structures; (3) examining the sensitivity of the encoder and decoder; (4) discussing different training strategies; and (5) discovering the repetition in a summary generation. The experimental results on prevalent MDS datasets and eleven evaluation metrics show the influence of document boundary separators, the granularity of different level features and different model training strategies. The results also reveal that the decoder exhibits greater sensitivity to noises compared to the encoder. This underscores the important role played by the decoder, suggesting a potential direction for future research in MDS. Furthermore, the experimental results indicate that the repetition problem in the generated summaries has correlations with the high uncertainty scores.