Generic resources are what you need: Style transfer tasks without task-specific parallel training data

TL;DR

This paper introduces a novel style transfer method that leverages generic resources and pre-trained models, achieving state-of-the-art results without task-specific parallel data by using iterative back-translation and synthetic data generation.

Contribution

It presents a new approach that outperforms existing methods by avoiding task-specific parallel data, utilizing generic resources and iterative back-translation with a pre-trained sequence-to-sequence model.

Findings

Outperforms existing unsupervised approaches on style transfer tasks

Effective use of generic paraphrases and synthetic pairs improves transfer quality

Highlights differences in task response to the proposed methodology

Abstract

Style transfer aims to rewrite a source text in a different target style while preserving its content. We propose a novel approach to this task that leverages generic resources, and without using any task-specific parallel (source-target) data outperforms existing unsupervised approaches on the two most popular style transfer tasks: formality transfer and polarity swap. In practice, we adopt a multi-step procedure which builds on a generic pre-trained sequence-to-sequence model (BART). First, we strengthen the model's ability to rewrite by further pre-training BART on both an existing collection of generic paraphrases, as well as on synthetic pairs created using a general-purpose lexical resource. Second, through an iterative back-translation approach, we train two models, each in a transfer direction, so that they can provide each other with synthetically generated pairs, dynamically in the training process. Lastly, we let our best reresulting model generate static synthetic pairs to be used in a supervised training regime. Besides methodology and state-of-the-art results, a core contribution of this work is a reflection on the nature of the two tasks we address, and how their differences are highlighted by their response to our approach.