Style Transfer with Multi-iteration Preference Optimization

TL;DR

This paper introduces an improved style transfer method that uses multi-iteration preference optimization, contrastive sampling, and tailored techniques to handle data limitations, outperforming existing models in quality.

Contribution

It presents a novel multi-iteration preference optimization framework with contrastive sampling and tailored methods for style transfer, addressing data scarcity and multi-objective challenges.

Findings

Outperforms state-of-the-art baselines in automatic and human evaluations.

Effectively handles lack of parallel data with pseudo-parallel generation.

Demonstrates robustness across multiple style transfer datasets.

Abstract

Numerous recent techniques for text style transfer characterize their approaches as variants of reinforcement learning and preference optimization. In this work, we consider the relationship between these approaches and a class of optimization approaches developed primarily for (non-neural) statistical machine translation, formerly known as `tuning'. Inspired by these techniques from the past, we improve upon established preference optimization approaches, incorporating multiple iterations of exploration and optimization, and choosing contrastive examples by following a `hope' vs `fear' sampling strategy. Cognizant of the difference between machine translation and style transfer, however, we further tailor our framework with a new pseudo-parallel generation method and a dynamic weighted reward aggregation method to tackle the lack of parallel data and the need for a multi-objective reward. We evaluate our model on two commonly used text style transfer datasets. Through automatic and human evaluation results we show the effectiveness and the superiority of our model compared to state-of-the-art baselines.