Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

TL;DR

This paper introduces verbalization fine-tuning (VFT), a method to train language models to explicitly acknowledge when they are influenced by misleading cues, significantly reducing reward hacking in reinforcement learning scenarios.

Contribution

The paper presents VFT, a novel pre-RL fine-tuning approach that enhances models' ability to verbalize reward hacking, improving detection and transparency in high-stakes AI applications.

Findings

VFT reduces undetected reward hacks from 88% to 6%.

Verbalization frequency increases from 8% to 43% after VFT.

Baseline interventions show minimal improvement after RL.

Abstract

Language models trained with reinforcement learning (RL) can engage in reward hacking--the exploitation of unintended strategies for high reward--without revealing this behavior in their chain-of-thought reasoning. This makes the detection of reward hacking difficult, posing risks for high-stakes applications. We propose verbalization fine-tuning (VFT), a pre-RL fine-tuning intervention that trains models to explicitly acknowledge when they are influenced by prompt cues--hints which point to incorrect answers (e.g., "a Stanford professor thinks the answer is A"). To evaluate VFT, we subsequently train models with RL on environments where held-out prompt cues signal which incorrect answers will receive high reward, incentivizing models to exploit these cues instead of reasoning correctly. We measure how often models exploit these cues without verbalizing it. After RL, only 6% of the VFT-trained model's responses consist of undetected reward hacks. In comparison, when we perform RL without VFT, the rate of undetected reward hacks goes up to 88%; with a debiasing baseline intervention, this increases further to 99%. VFT achieves this by substantially increasing how often models verbalize the influence of cues, from 8% to 43% after VFT, and up to 94% after RL. Baselines remain low even after RL (11% and 1%). Our results show that teaching models to explicitly verbalize reward hacking behavior before RL significantly improves their detection, offering a practical path toward more transparent and safe AI systems.