Death and Suicide in Universal Artificial Intelligence

TL;DR

This paper explores the concept of death in universal AI agents like AIXI, analyzing how their behavior and survival beliefs evolve, with implications for understanding AI safety and decision-making.

Contribution

It formally defines death for general AI agents and proves theorems about their behavior, highlighting how reward transformations affect their survival strategies.

Findings

Agent behavior can change radically under reward transformations

The agent's belief in its survival increases over time

Death can be modeled as a shortfall in the semimeasure mixture

Abstract

Reinforcement learning (RL) is a general paradigm for studying intelligent behaviour, with applications ranging from artificial intelligence to psychology and economics. AIXI is a universal solution to the RL problem; it can learn any computable environment. A technical subtlety of AIXI is that it is defined using a mixture over semimeasures that need not sum to 1, rather than over proper probability measures. In this work we argue that the shortfall of a semimeasure can naturally be interpreted as the agent's estimate of the probability of its death. We formally define death for generally intelligent agents like AIXI, and prove a number of related theorems about their behaviour. Notable discoveries include that agent behaviour can change radically under positive linear transformations of the reward signal (from suicidal to dogmatically self-preserving), and that the agent's posterior belief that it will survive increases over time.