Replacing Language Model for Style Transfer

TL;DR

This paper proposes the Replacing Language Model (RLM), a novel sequence-to-sequence framework for text style transfer that combines autoregressive and non-autoregressive methods to improve style transfer accuracy and control.

Contribution

The paper introduces RLM, a new style transfer approach that autoregressively replaces tokens with contextually similar spans generated by a non-autoregressive model, enhancing style transfer quality.

Findings

RLM outperforms existing style transfer baselines.

It effectively preserves local context and style content.

Token-level style-content disentanglement improves control.

Abstract

We introduce replacing language model (RLM), a sequence-to-sequence language modeling framework for text style transfer (TST). Our method autoregressively replaces each token of the source sentence with a text span that has a similar meaning but in the target style. The new span is generated via a non-autoregressive masked language model, which can better preserve the local-contextual meaning of the replaced token. This RLM generation scheme gathers the flexibility of autoregressive models and the accuracy of non-autoregressive models, which bridges the gap between sentence-level and word-level style transfer methods. To control the generation style more precisely, we conduct a token-level style-content disentanglement on the hidden representations of RLM. Empirical results on real-world text datasets demonstrate the effectiveness of RLM compared with other TST baselines. The code is at https://github.com/Linear95/RLM.