On the Sensitivity of Reward Inference to Misspecified Human Models

TL;DR

This paper investigates how inaccuracies in human behavior models affect reward inference in AI, revealing conditions under which reward errors can be controlled and demonstrating these findings through theoretical analysis and experiments.

Contribution

It provides a theoretical framework for understanding the impact of human model misspecification on reward inference and identifies conditions for bounded error, supported by empirical validation.

Findings

Small adversarial biases can cause large reward inference errors.

Under certain assumptions, reward error is linearly bounded by model error.

Empirical results confirm theoretical bounds in simulated and human data.

Abstract

Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.