Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

TL;DR

This paper introduces NAUS, a non-autoregressive unsupervised summarization method that uses search and training to generate summaries without parallel data, achieving state-of-the-art results and efficient inference.

Contribution

It presents a novel unsupervised summarization framework combining search-based pseudo-groundtruth generation with non-autoregressive Transformer training.

Findings

Achieves state-of-the-art performance in unsupervised summarization

Significantly improves inference efficiency

Enables explicit length-transfer summary generation

Abstract

Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation.