Language Detoxification with Attribute-Discriminative Latent Space

TL;DR

This paper introduces an efficient method for detoxifying language generated by Transformer models by projecting their latent space onto an attribute-discriminative space, significantly reducing toxicity while maintaining performance.

Contribution

The paper presents a novel attribute-discriminative latent space approach that improves detoxification efficiency and effectiveness in Transformer-based language models.

Findings

Outperforms baseline detoxification methods in quality

Reduces memory and computation overhead

Effective in language and dialogue generation tasks

Abstract

Transformer-based Language Models (LMs) have achieved impressive results on natural language understanding tasks, but they can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. To overcome this issue, a few text generation approaches aim to detoxify toxic texts using additional LMs or perturbations. However, previous methods require excessive memory, computations, and time which are serious bottlenecks in their real-world application. To address such limitations, we propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space. Specifically, we project the latent space of an original Transformer LM onto a discriminative latent space that well-separates texts by their attributes using a projection block and an attribute discriminator. This allows the LM to control the text generation to be non-toxic with minimal memory and computation overhead. We validate our model, Attribute-Discriminative Language Model (ADLM) on detoxified language and dialogue generation tasks, on which our method significantly outperforms baselines both in performance and efficiency.