Characterising Toxicity in Generative Large Language Models

TL;DR

This paper investigates how large language models produce toxic outputs when prompted and analyzes linguistic factors influencing such harmful responses, highlighting challenges in aligning models with human values.

Contribution

It provides a detailed characterization of toxicity in generative large language models and examines linguistic factors affecting toxic output generation.

Findings

LLMs can generate toxic responses when prompted.

Linguistic features influence the likelihood of toxicity.

Current safeguards can be bypassed with specific prompts.

Abstract

In recent years, the advent of the attention mechanism has significantly advanced the field of natural language processing (NLP), revolutionizing text processing and text generation. This has come about through transformer-based decoder-only architectures, which have become ubiquitous in NLP due to their impressive text processing and generation capabilities. Despite these breakthroughs, language models (LMs) remain susceptible to generating undesired outputs: inappropriate, offensive, or otherwise harmful responses. We will collectively refer to these as ``toxic'' outputs. Although methods like reinforcement learning from human feedback (RLHF) have been developed to align model outputs with human values, these safeguards can often be circumvented through carefully crafted prompts. Therefore, this paper examines the extent to which LLMs generate toxic content when prompted, as well as the linguistic factors -- both lexical and syntactic -- that influence the production of such outputs in generative models.