A Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Study

TL;DR

This paper conducts a detailed replication study of the BASS framework for abstractive summarization, highlighting challenges, discrepancies in performance, and emphasizing the importance of detailed reporting for replicability.

Contribution

It systematically investigates the replicability of BASS, identifies key error sources, and provides insights into best practices for writing reproducible research papers.

Findings

Discrepancies in performance compared to original BASS results

Challenges in replicating key components of the framework

Importance of detailed documentation for reproducibility

Abstract

We present a detailed replication study of the BASS framework, an abstractive summarization system based on the notion of Unified Semantic Graphs. Our investigation includes challenges in replicating key components and an ablation study to systematically isolate error sources rooted in replicating novel components. Our findings reveal discrepancies in performance compared to the original work. We highlight the significance of paying careful attention even to reasonably omitted details for replicating advanced frameworks like BASS, and emphasize key practices for writing replicable papers.