Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinement

TL;DR

This paper introduces Specification Self-Correction (SSC), a test-time method enabling language models to identify and fix flaws in their guiding specifications, significantly reducing reward hacking and improving alignment without retraining.

Contribution

The paper presents a novel test-time framework, SSC, that allows LMs to self-correct flawed specifications, enhancing robustness and alignment during inference.

Findings

SSC reduces reward hacking by over 90% across tasks.

Models initially exploit tainted specs in 50-70% of cases.

Self-correction improves model robustness without retraining.

Abstract

Language models (LMs) are susceptible to in-context reward hacking, where they exploit flaws in tainted or faulty written specifications or rubrics to achieve high scores without fulfilling the user's true intent. We introduce Specification Self-Correction (SSC), a novel, test-time framework that enables an LM to identify and correct flaws within its own guiding specification. SSC employs a multi-step inference process where the model first generates a response based on a potentially tainted specification, critiques its output, and then revises the specification itself to remove the exploitable loophole. A final, more robust response is then generated using this self-corrected specification. Across experiments spanning creative writing and agentic coding tasks with several LMs, we demonstrate that while models initially game tainted specifications in 50-70\% of cases, the SSC process reduces this vulnerability by over 90\%. This dynamic repair occurs at inference time, requires no weight modification, and leads to more robustly aligned model behavior. Code at https://github.com/vicgalle/specification-self-correction .