Toxicity Detection for Free

TL;DR

This paper introduces MULI, a novel toxicity detection method leveraging LLMs' own response logits, outperforming existing detectors in accuracy and efficiency for identifying toxic prompts.

Contribution

We propose MULI, a new toxicity detection approach that uses LLMs' first response token logits, achieving superior performance over state-of-the-art methods.

Findings

MULI outperforms SOTA detectors on multiple metrics.

The first response token logits effectively distinguish toxic from benign prompts.

The method is robust and cost-effective for real-world applications.

Abstract

Current LLMs are generally aligned to follow safety requirements and tend to refuse toxic prompts. However, LLMs can fail to refuse toxic prompts or be overcautious and refuse benign examples. In addition, state-of-the-art toxicity detectors have low TPRs at low FPR, incurring high costs in real-world applications where toxic examples are rare. In this paper, we introduce Moderation Using LLM Introspection (MULI), which detects toxic prompts using the information extracted directly from LLMs themselves. We found we can distinguish between benign and toxic prompts from the distribution of the first response token's logits. Using this idea, we build a robust detector of toxic prompts using a sparse logistic regression model on the first response token logits. Our scheme outperforms SOTA detectors under multiple metrics.