Convergence of Expected Utility for Universal AI

TL;DR

This paper investigates the convergence properties of expected utility in universal AI, showing that with unbounded utility functions, the expected utility becomes undefined, highlighting fundamental limitations in decision-making under uncertainty.

Contribution

It demonstrates that for universal AI models with unbounded utility functions, the expected utility does not converge, revealing a critical theoretical limitation.

Findings

Expected utility is undefined for unbounded utility functions in universal AI.

Convergence issues arise in repeated interactions with uncertain environments.

Highlights fundamental limitations in the theoretical foundations of universal AI.

Abstract

We consider a sequence of repeated interactions between an agent and an environment. Uncertainty about the environment is captured by a probability distribution over a space of hypotheses, which includes all computable functions. Given a utility function, we can evaluate the expected utility of any computational policy for interaction with the environment. After making some plausible assumptions (and maybe one not-so-plausible assumption), we show that if the utility function is unbounded, then the expected utility of any policy is undefined.