StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer

TL;DR

StylePTB introduces a comprehensive benchmark for fine-grained and compositional text style transfer, highlighting current models' limitations and fostering future research in controllable and disentangled text generation.

Contribution

The paper presents StylePTB, a large-scale benchmark with diverse fine-grained stylistic changes and compositions, enabling detailed evaluation of style transfer methods.

Findings

Existing methods struggle with fine-grained style modeling.

Models have difficulty composing multiple stylistic changes.

StylePTB reveals significant challenges for current style transfer techniques.

Abstract

Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, StylePTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text, as well as (2) compositions of multiple transfers which allow modeling of fine-grained stylistic changes as building blocks for more complex, high-level transfers. By benchmarking existing methods on StylePTB, we find that they struggle to model fine-grained changes and have an even more difficult time composing multiple styles. As a result, StylePTB brings novel challenges that we hope will encourage future research in controllable text style transfer, compositional models, and learning disentangled representations. Solving these challenges would present important steps towards controllable text generation.