Don't lose the message while paraphrasing: A study on content preserving style transfer

TL;DR

This paper evaluates style transfer models focusing on content preservation in goal-oriented dialogues, introducing a new dataset with semantic slots and improving an existing method for better factual consistency.

Contribution

It presents a new dataset with semantic annotations for style transfer, and enhances the LEWIS method to better preserve content in goal-oriented dialogue paraphrasing.

Findings

The new dataset enables precise evaluation of content preservation.

Modified LEWIS outperforms baseline models in content retention.

Content preservation is critical for real-world style transfer applications.

Abstract

Text style transfer techniques are gaining popularity in natural language processing allowing paraphrasing text in the required form: from toxic to neural, from formal to informal, from old to the modern English language, etc. Solving the task is not sufficient to generate some neural/informal/modern text, but it is important to preserve the original content unchanged. This requirement becomes even more critical in some applications such as style transfer of goal-oriented dialogues where the factual information shall be kept to preserve the original message, e.g. ordering a certain type of pizza to a certain address at a certain time. The aspect of content preservation is critical for real-world applications of style transfer studies, but it has received little attention. To bridge this gap we perform a comparison of various style transfer models on the example of the formality transfer domain. To perform a study of the content preservation abilities of various style transfer methods we create a parallel dataset of formal vs. informal task-oriented dialogues. The key difference between our dataset and the existing ones like GYAFC [17] is the presence of goal-oriented dialogues with predefined semantic slots essential to be kept during paraphrasing, e.g. named entities. This additional annotation allowed us to conduct a precise comparative study of several state-of-the-art techniques for style transfer. Another result of our study is a modification of the unsupervised method LEWIS [19] which yields a substantial improvement over the original method and all evaluated baselines on the proposed task.