Improving the Robustness of Summarization Models by Detecting and Removing Input Noise

TL;DR

This paper investigates how input noise affects summarization models' performance and introduces a simple, training-free method to detect and remove noise, significantly improving robustness across datasets.

Contribution

The paper provides a comprehensive empirical analysis of noise impact on summarization and proposes a lightweight, effective noise removal technique without additional training.

Findings

Input noise can reduce ROUGE-1 scores by up to 12 points.

The proposed noise detection method recovers up to 11 ROUGE-1 points.

The approach works across different datasets and model sizes.

Abstract

The evaluation of abstractive summarization models typically uses test data that is identically distributed as training data. In real-world practice, documents to be summarized may contain input noise caused by text extraction artifacts or data pipeline bugs. The robustness of model performance under distribution shift caused by such noise is relatively under-studied. We present a large empirical study quantifying the sometimes severe loss in performance (up to 12 ROUGE-1 points) from different types of input noise for a range of datasets and model sizes. We then propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any extra training, auxiliary models, or even prior knowledge of the type of noise. Our proposed approach effectively mitigates the loss in performance, recovering a large fraction of the performance drop, sometimes as large as 11 ROUGE-1 points.