Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training

TL;DR

This paper introduces a large-scale dataset for complex text style transfer and demonstrates that small models with latent attribute pre-training can outperform larger models in this task, offering a privacy-preserving and cost-effective solution.

Contribution

The work presents a new large-scale dataset for complex style transfer and a novel approach using small models with implicit style pre-training, achieving state-of-the-art results.

Findings

Small models outperform larger models in style transfer tasks.

Automated evaluation correlates well with human judgments.

Pre-training enhances style transfer effectiveness.

Abstract

In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact. While large language models (LLM) have shown promise in complex text style transfer, they have drawbacks such as data privacy concerns, network instability, and high deployment costs. To address these issues, we explore the effectiveness of small models (less than T5-3B) with implicit style pre-training through contrastive learning. We also propose a method for automated evaluation of text generation quality based on alignment with human evaluations using ChatGPT. Finally, we compare our approach with existing methods and show that our model achieves state-of-art performances of few-shot text style transfer models.