Transductive Learning for Abstractive News Summarization

TL;DR

This paper introduces a transductive learning approach for abstractive news summarization that fine-tunes models on test inputs, leading to improved ROUGE scores and more coherent, abstract summaries without altering the original architecture.

Contribution

It is the first to apply transductive learning to summarization, enhancing performance by fine-tuning on test set inputs and pseudo summaries, with state-of-the-art results.

Findings

Improved ROUGE-L scores on CNN/DM and NYT datasets.

Enhanced summary coherence and abstractiveness.

Effective transductive fine-tuning without architectural changes.

Abstract

Pre-trained and fine-tuned news summarizers are expected to generalize to news articles unseen in the fine-tuning (training) phase. However, these articles often contain specifics, such as new events and people, a summarizer could not learn about in training. This applies to scenarios such as a news publisher training a summarizer on dated news and summarizing incoming recent news. In this work, we explore the first application of transductive learning to summarization where we further fine-tune models on test set inputs. Specifically, we construct pseudo summaries from salient article sentences and input randomly masked articles. Moreover, this approach is also beneficial in the fine-tuning phase, where we jointly predict extractive pseudo references and abstractive gold summaries in the training set. We show that our approach yields state-of-the-art results on CNN/DM and NYT datasets, improving ROUGE-L by 1.05 and 0.74, respectively. Importantly, our approach does not require any changes of the original architecture. Moreover, we show the benefits of transduction from dated to more recent CNN news. Finally, through human and automatic evaluation, we demonstrate improvements in summary abstractiveness and coherence.