Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars

TL;DR

This paper introduces a novel span prediction-based paraphrasing method extending GPT-2, employing masking techniques to improve semantic preservation and generate diverse paraphrases without retraining for each dataset.

Contribution

It proposes first-order and second-order masking techniques for paraphrase generation, enhancing semantic fidelity and diversity while avoiding retraining for new datasets.

Findings

Outperforms baselines in semantic preservation

Enables diverse paraphrases through masking control

Template selection methods are equally effective

Abstract

Paraphrase generation is a long-standing problem and serves an essential role in many natural language processing problems. Despite some encouraging results, recent methods either confront the problem of favoring generic utterance or need to retrain the model from scratch for each new dataset. This paper presents a novel approach to paraphrasing sentences, extended from the GPT-2 model. We develop a template masking technique, named first-order masking, to masked out irrelevant words in exemplars utilizing POS taggers. So that, the paraphrasing task is changed to predicting spans in masked templates. Our proposed approach outperforms competitive baselines, especially in the semantic preservation aspect. To prevent the model from being biased towards a given template, we introduce a technique, referred to as second-order masking, which utilizes Bernoulli distribution to control the visibility of the first-order-masked template's tokens. Moreover, this technique allows the model to provide various paraphrased sentences in testing by adjusting the second-order-masking level. For scale-up objectives, we compare the performance of two alternatives template-selection methods, which shows that they were equivalent in preserving semantic information.