The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret

TL;DR

This paper demonstrates that low reward model error during training does not necessarily lead to low regret during policy deployment due to distributional shifts, highlighting the need for better evaluation methods.

Contribution

The paper provides a mathematical analysis linking reward model error to regret, revealing limitations of current reward learning approaches under distributional shifts.

Findings

Low expected test error guarantees low worst-case regret

Error-regret mismatch can occur despite low training error

Policy regularization does not fully prevent error-regret mismatch

Abstract

In reinforcement learning, specifying reward functions that capture the intended task can be very challenging. Reward learning aims to address this issue by learning the reward function. However, a learned reward model may have a low error on the data distribution, and yet subsequently produce a policy with large regret. We say that such a reward model has an error-regret mismatch. The main source of an error-regret mismatch is the distributional shift that commonly occurs during policy optimization. In this paper, we mathematically show that a sufficiently low expected test error of the reward model guarantees low worst-case regret, but that for any fixed expected test error, there exist realistic data distributions that allow for error-regret mismatch to occur. We then show that similar problems persist even when using policy regularization techniques, commonly employed in methods such as RLHF. We hope our results stimulate the theoretical and empirical study of improved methods to learn reward models, and better ways to measure their quality reliably.