When Can Proxies Improve the Sample Complexity of Preference Learning?

TL;DR

This paper investigates conditions under which proxy feedback can enhance the sample efficiency of learning true policies, providing theoretical insights and practical guidelines for leveraging proxy data in preference learning.

Contribution

It establishes sufficient conditions on proxy feedback that guarantee improved sample complexity in learning true policies, guiding data collection and model design.

Findings

Proposes conditions for proxy feedback to improve sample complexity.

Provides a parameterization for LLMs to achieve better learning efficiency.

Suggests adaptations of existing architectures for improved sample complexity.

Abstract

We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity.