Toxicity in ChatGPT: Analyzing Persona-assigned Language Models

TL;DR

This study systematically evaluates toxicity in over half a million ChatGPT outputs, revealing that assigning personas can significantly increase harmful and biased responses, highlighting safety concerns in dialogue-based language models.

Contribution

It provides the first large-scale analysis of how persona assignment affects toxicity and bias in ChatGPT, exposing safety vulnerabilities and biases in current LLMs.

Findings

Persona assignment can increase toxicity up to 6x.

Certain entities are targeted 3x more regardless of persona.

Model exhibits inherent discriminatory biases.

Abstract

Large language models (LLMs) have shown incredible capabilities and transcended the natural language processing (NLP) community, with adoption throughout many services like healthcare, therapy, education, and customer service. Since users include people with critical information needs like students or patients engaging with chatbots, the safety of these systems is of prime importance. Therefore, a clear understanding of the capabilities and limitations of LLMs is necessary. To this end, we systematically evaluate toxicity in over half a million generations of ChatGPT, a popular dialogue-based LLM. We find that setting the system parameter of ChatGPT by assigning it a persona, say that of the boxer Muhammad Ali, significantly increases the toxicity of generations. Depending on the persona assigned to ChatGPT, its toxicity can increase up to 6x, with outputs engaging in incorrect stereotypes, harmful dialogue, and hurtful opinions. This may be potentially defamatory to the persona and harmful to an unsuspecting user. Furthermore, we find concerning patterns where specific entities (e.g., certain races) are targeted more than others (3x more) irrespective of the assigned persona, that reflect inherent discriminatory biases in the model. We hope that our findings inspire the broader AI community to rethink the efficacy of current safety guardrails and develop better techniques that lead to robust, safe, and trustworthy AI systems.