Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions

TL;DR

This paper explores combining attention masking and Large Language Models for unsupervised text style transfer, introducing multi-way interactions that improve style strength, content preservation, and fluency, surpassing existing methods including supervised systems.

Contribution

It proposes four multi-way interaction methods to effectively combine attention masking and LLMs for improved unsupervised text style transfer performance.

Findings

Achieves new state-of-the-art results on Yelp-clean and Amazon-clean datasets.

Simple prompting with attention masking surpasses other systems.

Improvements include better style strength, content preservation, and fluency.

Abstract

Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP), aiming to transfer one stylistic aspect of a sentence into another style without changing its semantics, syntax, or other attributes. This task is especially challenging given the intrinsic lack of parallel text pairings. Among existing methods for UTST tasks, attention masking approach and Large Language Models (LLMs) are deemed as two pioneering methods. However, they have shortcomings in generating unsmooth sentences and changing the original contents, respectively. In this paper, we investigate if we can combine these two methods effectively. We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from LLMs to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency. Experiments also demonstrate that simply conducting prompting followed by attention masking-based revision can consistently surpass the other systems, including supervised text style transfer systems. On Yelp-clean and Amazon-clean datasets, it improves the previously best mean metric by 0.5 and 3.0 absolute percentages respectively, and achieves new SOTA results.