Challenges in Detoxifying Language Models

TL;DR

This paper critically examines toxicity mitigation in large language models, revealing that current strategies may reduce bias but also diminish model coverage and can conflict with human judgments, highlighting evaluation challenges.

Contribution

It provides a comprehensive analysis of toxicity mitigation strategies, comparing automatic and human evaluations, and discusses their impact on model bias and quality.

Findings

Basic interventions improve automatic toxicity metrics

Toxicity mitigation reduces model coverage for marginalized groups

Human judgments often differ from automatic toxicity scores

Abstract

Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of LM toxicity. We critically discuss this approach, evaluate several toxicity mitigation strategies with respect to both automatic and human evaluation, and analyze consequences of toxicity mitigation in terms of model bias and LM quality. We demonstrate that while basic intervention strategies can effectively optimize previously established automatic metrics on the RealToxicityPrompts dataset, this comes at the cost of reduced LM coverage for both texts about, and dialects of, marginalized groups. Additionally, we find that human raters often disagree with high automatic toxicity scores after strong toxicity reduction interventions -- highlighting further the nuances involved in careful evaluation of LM toxicity.