Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

TL;DR

This paper challenges the idea that scalar reward functions are sufficient for intelligence, arguing for multi-objective models to better capture biological and computational complexity and ensure safety in artificial intelligence development.

Contribution

It critiques the scalar reward assumption in AI, advocates for multi-objective reward models, and discusses safety concerns in artificial general intelligence.

Findings

Scalar rewards are insufficient for complex intelligence.

Multi-objective reward models better reflect biological intelligence.

Scalar reward-based AI poses safety and ethical risks.

Abstract

The recent paper `"Reward is Enough" by Silver, Singh, Precup and Sutton posits that the concept of reward maximisation is sufficient to underpin all intelligence, both natural and artificial. We contest the underlying assumption of Silver et al. that such reward can be scalar-valued. In this paper we explain why scalar rewards are insufficient to account for some aspects of both biological and computational intelligence, and argue in favour of explicitly multi-objective models of reward maximisation. Furthermore, we contend that even if scalar reward functions can trigger intelligent behaviour in specific cases, it is still undesirable to use this approach for the development of artificial general intelligence due to unacceptable risks of unsafe or unethical behaviour.