A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens

TL;DR

This paper introduces a novel safety mitigation method for large language models by training them to insert a red flag token when harmful content is detected, enabling explicit harmfulness recognition with minimal utility loss.

Contribution

The authors propose augmenting LLMs with a red flag token to explicitly represent harmfulness, leveraging in-context learning for safety without degrading task performance.

Findings

Model can learn to insert red flag tokens to indicate harmful content.

In-context learning enables the model to initiate reflective reasoning upon flagging.

Method is orthogonal to existing safety techniques and easier to evaluate.

Abstract

Many safety post-training methods for large language models (LLMs) are designed to modify the model's behaviour from producing unsafe answers to issuing refusals. However, such distribution shifts are often brittle and degrade performance on desirable tasks. To address these pitfalls, we propose augmenting the model's vocabulary with a special red flag token, and training the model to insert this token whenever harmful content is generated or imminent. This approach enables the model to explicitly learn the concept of harmfulness in its representations, with minimal impact on utility due to the marginal change in the generated distribution of natural language. Moreover, because the token is embedded in the model's vocabulary, we can naturally leverage the LLMs' generalization capabilities, such as in-context learning (ICL) and out-of-distribution generalization to languages that are not formally supported (e.g., Japanese for Llama3). In particular, we demonstrate that through ICL alone, the model can learn to initiate reflective reasoning upon generating the red flag token at inference, which steers the response away from harmful continuations or enables self-correction when the flag is raised falsely. This approach is orthogonal and complementary to existing safety technique (such as safety classifiers or standard safety training) and easier to evaluate in comparison to natural language refusals, as it does not require a human or automated judge to assess the harmlessness of the answers.