Multi-document Summarization: A Comparative Evaluation

TL;DR

This paper evaluates state-of-the-art multi-document summarization models across diverse datasets and domains, highlighting the LED model's superior performance on complex datasets and providing insights for future research directions.

Contribution

It offers a comprehensive comparison of MDS models on varied datasets, identifying the strengths and limitations of current models and guiding future advancements in the field.

Findings

LED outperforms PRIMERA and PEGASUS on MS$^2$ dataset

ROUGE scores used for performance evaluation

Insights into models' domain-specific strengths and weaknesses

Abstract

This paper is aimed at evaluating state-of-the-art models for Multi-document Summarization (MDS) on different types of datasets in various domains and investigating the limitations of existing models to determine future research directions. To address this gap, we conducted an extensive literature review to identify state-of-the-art models and datasets. We analyzed the performance of PRIMERA and PEGASUS models on BigSurvey-MDS and MS$^2$ datasets, which posed unique challenges due to their varied domains. Our findings show that the General-Purpose Pre-trained Model LED outperforms PRIMERA and PEGASUS on the MS$^2$ dataset. We used the ROUGE score as a performance metric to evaluate the identified models on different datasets. Our study provides valuable insights into the models' strengths and weaknesses, as well as their applicability in different domains. This work serves as a reference for future MDS research and contributes to the development of accurate and robust models which can be utilized on demanding datasets with academically and/or scientifically complex data as well as generalized, relatively simple datasets.