Unsupervised Summarization Re-ranking

TL;DR

This paper introduces an unsupervised re-ranking method for summarization models like PEGASUS and ChatGPT, significantly improving their zero-shot summarization performance across multiple benchmarks.

Contribution

It proposes a novel unsupervised re-ranking approach that enhances the quality of candidate summaries, narrowing the gap with supervised models.

Findings

Up to 7.27% improvement in ROUGE scores for PEGASUS.

Up to 6.86% improvement in ROUGE scores for ChatGPT.

Average 7.51% gain across 30 zero-shot transfer setups.

Abstract

With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks. However, the performance of such unsupervised models still lags significantly behind their supervised counterparts. Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output. In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models. Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).