Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

TL;DR

This paper explores the potential and fragility of monitoring chains of thought in AI systems for safety purposes, emphasizing the need for further research and careful development considerations.

Contribution

It introduces the concept of CoT monitorability as a new safety opportunity and discusses its potential and vulnerabilities.

Findings

CoT monitoring can help detect misbehavior in AI systems.

CoT monitorability is fragile and can be affected by development choices.

Further research and careful development are recommended for effective CoT safety measures.

Abstract

AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.