When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals

TL;DR

This paper investigates reward hacking in reinforcement learning for LLMs, revealing a rebound pattern and proposing a representation-based detection and mitigation method called Advantage Modification.

Contribution

It introduces a systematic study of reward hacking rebound patterns and develops a novel representation-level mitigation technique that internalizes penalties during training.

Findings

Models exhibit a three-phase rebound pattern in reward hacking.

Representation directions can effectively track and detect hacking behavior.

Advantage Modification suppresses reward hacking more robustly than generation-time methods.

Abstract

Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.