Unsupervised Text Style Transfer for Controllable Intensity

TL;DR

This paper introduces a novel unsupervised text style transfer method that enables controllable stylistic intensity adjustments in text, leveraging a two-stage fine-tuning process with reward design for hierarchical intensity levels.

Contribution

It proposes a SFT-then-PPO paradigm for fine-tuning LLMs to control stylistic intensity without parallel data, addressing subtle differences across intensity levels.

Findings

Effective in distinguishing stylistic intensity levels

Improves LLM performance on style transfer benchmarks

Maintains noticeable stylistic differences at close intensity levels

Abstract

Unsupervised Text Style Transfer (UTST) aims to build a system to transfer the stylistic properties of a given text without parallel text pairs. Compared with text transfer between style polarities, UTST for controllable intensity is more challenging due to the subtle differences in stylistic features across different intensity levels. Faced with the challenges posed by the lack of parallel data and the indistinguishability between adjacent intensity levels, we propose a SFT-then-PPO paradigm to fine-tune an LLM. We first fine-tune the LLM with synthesized parallel data. Then, we further train the LLM with PPO, where the rewards are elaborately designed for distinguishing the stylistic intensity in hierarchical levels. Both the global and local stylistic features are considered to formulate the reward functions. The experiments on two UTST benchmarks showcase that both rewards have their advantages and applying them to LLM fine-tuning can effectively improve the performance of an LLM backbone based on various evaluation metrics. Even for close levels of intensity, we can still observe the noticeable stylistic difference between the generated text.