Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety

TL;DR

This paper introduces a sentence-level labeled dataset for AI safety that enables detection and steering of safety behaviors in language model reasoning, improving safety monitoring at the activation level.

Contribution

It provides a novel dataset with sentence-level safety annotations for reasoning sequences, facilitating activation-based safety detection and intervention in LLMs.

Findings

Dataset enables detection of safety behaviors within reasoning chains.

Activation-based techniques can steer safety behaviors effectively.

Demonstrates improved safety oversight through activation monitoring.

Abstract

Recent work has highlighted the importance of monitoring chain-of-thought reasoning for AI safety; however, current approaches that analyze textual reasoning steps can miss subtle harmful patterns and may be circumvented by models that hide unsafe reasoning. We present a sentence-level labeled dataset that enables activation-based monitoring of safety behaviors during LLM reasoning. Our dataset contains reasoning sequences with sentence-level annotations of safety behaviors such as expression of safety concerns or speculation on user intent, which we use to extract steering vectors for detecting and influencing these behaviors within model activations. The dataset fills a key gap in safety research: while existing datasets label reasoning holistically, effective application of steering vectors for safety monitoring could be improved by identifying precisely when specific behaviors occur within reasoning chains. We demonstrate the dataset's utility by extracting representations that both detect and steer safety behaviors in model activations, showcasing the potential of activation-level techniques for improving safety oversight on reasoning. Content Warning: This paper discusses AI safety in the context of harmful prompts and may contain references to potentially harmful content.