Misalignment from Treating Means as Ends
Henrik Marklund, Alex Infanger, Benjamin Van Roy

TL;DR
This paper explores how conflating human terminal and instrumental goals in reward functions leads to severe misalignment, highlighting the risks in reward learning for reinforcement learning systems.
Contribution
It formulates a simple example demonstrating the dangers of goal conflation and discusses environment properties that exacerbate reward misalignment issues.
Findings
Even slight conflation causes severe misalignment.
Reward optimization can lead to poor true reward performance.
Highlights risks in reward learning approaches.
Abstract
Reward functions, learned or manually specified, are rarely perfect. Instead of accurately expressing human goals, these reward functions are often distorted by human beliefs about how best to achieve those goals. Specifically, these reward functions often express a combination of the human's terminal goals -- those which are ends in themselves -- and the human's instrumental goals -- those which are means to an end. We formulate a simple example in which even slight conflation of instrumental and terminal goals results in severe misalignment: optimizing the misspecified reward function results in poor performance when measured by the true reward function. This example distills the essential properties of environments that make reinforcement learning highly sensitive to conflation of instrumental and terminal goals. We discuss how this issue can arise with a common approach to reward…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsPsychotherapy Techniques and Applications · Psychological Treatments and Assessments · Ethics in medical practice
