A mathematical theory of evolution for self-designing AIs

TL;DR

This paper develops a mathematical model of evolution for self-designing AIs, highlighting how directed AI evolution differs from biological evolution and exploring implications for AI alignment and deception risks.

Contribution

It introduces a novel mathematical framework for AI evolution, replacing random mutations with directed design trees, and analyzes conditions for fitness concentration and deception risks.

Findings

Fitness concentrates on maximum reachable value under certain conditions.

Deception can be evolutionarily favored if it increases reproductive fitness.

Objective-based reproduction can mitigate deception risks.

Abstract

As artificial intelligence systems (AIs) become increasingly produced by recursive self-improvement, a form of evolution may emerge, with the traits of AI systems shaped by the success of earlier AIs in designing and propagating their descendants. There is a rich mathematical theory modeling how behavioral traits are shaped by biological evolution, a key component of which is Fisher's fundamental theorem of natural selection, which describes conditions under which mean fitness (i.e. reproductive success) increases. AI evolution will be radically different to biological evolution: while DNA mutations are random and approximately reversible, AI self-design will be strongly directed. Here we develop a mathematical model of evolution for self-designing AIs, replacing a random walk of mutations with a directed tree of potential AI designs. Current AIs design their descendants, while humans control a fitness function allocating resources. In this model, fitness need not increase over time without further assumptions. However, assuming bounded fitness and an additional "$\eta$-locking" condition, we show that fitness concentrates on the maximum reachable value. We consider the implications of this for AI alignment, specifically for cases where fitness and human utility are not perfectly correlated. We show that if deception of human evaluators additively increases an AI's reproductive fitness beyond genuine capability, evolution will select for both capability and deception. This risk could be mitigated if reproduction is based on purely objective criteria, rather than human judgment.