Teaching Models to Understand (but not Generate) High-risk Data

TL;DR

This paper introduces SLUNG, a pre-training method that helps language models understand high-risk content without generating it, improving safety and comprehension while maintaining output quality.

Contribution

SLUNG is a novel pre-training paradigm that enables models to understand high-risk data without learning to generate harmful content.

Findings

SLUNG improves models' ability to recognize toxic content.

SLUNG does not increase the toxicity of generated outputs.

Models trained with SLUNG better understand high-risk data without generating it.

Abstract

Language model developers typically filter out high-risk content -- such as toxic or copyrighted text -- from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models' ability to recognize and appropriately respond to harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it is forced to understand the high-risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e.g., toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.