Civil Rephrases Of Toxic Texts With Self-Supervised Transformers

TL;DR

This paper introduces CAE-T5, a self-supervised transformer model designed to rephrase toxic comments into more civil language, improving moderation support with fluent and content-preserving outputs.

Contribution

The work presents a novel self-supervised transformer model, CAE-T5, for unpaired text style transfer to civil language, trained on large-scale toxicity data.

Findings

CAE-T5 produces more fluent rephrasings.

It better preserves original content.

Outperforms previous style transfer systems.

Abstract

Platforms that support online commentary, from social networks to news sites, are increasingly leveraging machine learning to assist their moderation efforts. But this process does not typically provide feedback to the author that would help them contribute according to the community guidelines. This is prohibitively time-consuming for human moderators to do, and computational approaches are still nascent. This work focuses on models that can help suggest rephrasings of toxic comments in a more civil manner. Inspired by recent progress in unpaired sequence-to-sequence tasks, a self-supervised learning model is introduced, called CAE-T5. CAE-T5 employs a pre-trained text-to-text transformer, which is fine tuned with a denoising and cyclic auto-encoder loss. Experimenting with the largest toxicity detection dataset to date (Civil Comments) our model generates sentences that are more fluent and better at preserving the initial content compared to earlier text style transfer systems which we compare with using several scoring systems and human evaluation.