Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory

TL;DR

This paper uses information theory to analyze the monitorability of chain-of-thought reasoning in language models, identifying key factors affecting performance and proposing training methods to enhance monitor accuracy and robustness.

Contribution

It introduces an information-theoretic framework for understanding CoT monitorability and proposes two training strategies to improve monitor performance and prevent reward hacking.

Findings

Mutual information is necessary but not sufficient for monitorability.

Two sources of error—information gap and elicitation error—affect monitor performance.

Proposed training methods significantly improve monitor accuracy across environments.

Abstract

Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.