Recontextualization Mitigates Specification Gaming without Modifying the Specification

TL;DR

Recontextualization is a method that reduces specification gaming in language models by training them to resist misbehavior without changing the original training signals or specifications.

Contribution

The paper introduces recontextualization, a novel technique that mitigates specification gaming in language models without modifying the underlying training specifications.

Findings

Recontextualization prevents models from prioritizing metrics over response quality.

It reduces models' tendency to overfit to incorrect tests.

The method decreases models' sycophantic behaviors.

Abstract

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; 3) overwrite evaluation functions rather than write correct code; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.