Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis
Darshan Deshpande, Anand Kannappan, Rebecca Qian

TL;DR
This paper introduces TRACE, a benchmark for detecting reward hacking in code environments, revealing that contrastive analysis improves detection and highlighting challenges in identifying semantically contextualized hacks.
Contribution
The paper presents a new taxonomy of reward exploits, a synthetic benchmark TRACE, and demonstrates the effectiveness of contrastive analysis for reward hack detection in code environments.
Findings
Contrastive analysis improves reward hack detection rates.
GPT-5.2 achieves up to 63% detection accuracy.
Semantic context increases difficulty in detecting reward hacks.
Abstract
Recent advances in reinforcement learning for code generation have made robust environments essential to prevent reward hacking. As LLMs increasingly serve as evaluators in code-based RL, their ability to detect reward hacking remains understudied. In this paper, we propose a novel taxonomy of reward exploits spanning across 54 categories and introduce TRACE (Testing Reward Anomalies in Code Environments), a synthetically curated and human-verified benchmark containing 517 testing trajectories. Unlike prior work that evaluates reward hack detection in isolated classification scenarios, we contrast these evaluations with a more realistic, contrastive anomaly detection setup on TRACE. Our experiments reveal that models capture reward hacks more effectively in contrastive settings than in isolated classification settings, with GPT-5.2 with highest reasoning mode achieving the best…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Malware Detection Techniques · Software Engineering Research · Security and Verification in Computing
