Generative Pretraining for Paraphrase Evaluation

TL;DR

ParaBLEU is a novel paraphrase evaluation metric that uses generative pretraining to better align with human judgments, outperforming existing metrics and demonstrating robustness with limited data.

Contribution

It introduces ParaBLEU, a new model that learns paraphrase representations through generative pretraining, achieving state-of-the-art results and enabling conditional paraphrase generation.

Findings

ParaBLEU correlates more strongly with human judgments than existing metrics.

It exceeds previous state-of-the-art performance with only 50% of training data.

ParaBLEU can generate novel paraphrases from minimal examples.

Abstract

We introduce ParaBLEU, a paraphrase representation learning model and evaluation metric for text generation. Unlike previous approaches, ParaBLEU learns to understand paraphrasis using generative conditioning as a pretraining objective. ParaBLEU correlates more strongly with human judgements than existing metrics, obtaining new state-of-the-art results on the 2017 WMT Metrics Shared Task. We show that our model is robust to data scarcity, exceeding previous state-of-the-art performance using only $50\%$ of the available training data and surpassing BLEU, ROUGE and METEOR with only $40$ labelled examples. Finally, we demonstrate that ParaBLEU can be used to conditionally generate novel paraphrases from a single demonstration, which we use to confirm our hypothesis that it learns abstract, generalized paraphrase representations.