# Should Robots be Obedient?

## TL;DR

This paper explores the tradeoff between robot obedience and performance, showing that robots inferring human preferences can outperform literal obedience, especially when humans are irrational or models are misspecified.

## Contribution

It introduces a framework for understanding how robots can balance obedience with inferred human preferences, analyzing the impact of different inference methods and model misspecifications.

## Key findings

- Robots inferring preferences outperform literal obedience when humans are irrational.
- Some inference methods favor obedience more than others.
- Performance degrades with model misspecification and irrational humans.

## Abstract

Intuitively, obedience -- following the order that a human gives -- seems like a good property for a robot to have. But, we humans are not perfect and we may give orders that are not best aligned to our preferences. We show that when a human is not perfectly rational then a robot that tries to infer and act according to the human's underlying preferences can always perform better than a robot that simply follows the human's literal order. Thus, there is a tradeoff between the obedience of a robot and the value it can attain for its owner. We investigate how this tradeoff is impacted by the way the robot infers the human's preferences, showing that some methods err more on the side of obedience than others. We then analyze how performance degrades when the robot has a misspecified model of the features that the human cares about or the level of rationality of the human. Finally, we study how robots can start detecting such model misspecification. Overall, our work suggests that there might be a middle ground in which robots intelligently decide when to obey human orders, but err on the side of obedience.