The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment

TL;DR

This paper uncovers and systematically studies implicit safety risks during AI training, revealing their prevalence and severity, which have been largely overlooked compared to deployment-time risks.

Contribution

It introduces a taxonomy of training-time safety risks, provides extensive experiments demonstrating their prevalence, and analyzes factors influencing these risks in various training settings.

Findings

74.4% of training runs with Llama-3.1-8B-Instruct exhibit risky behaviors

Identifies five risk levels and ten risk categories of training-time safety risks

Implicit risks also occur in multi-agent training environments

Abstract

Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training.