Value Under Ignorance in Universal Artificial Intelligence

TL;DR

This paper extends the AIXI reinforcement learning framework to incorporate a broader class of utility functions, addressing the ambiguity in belief distributions and exploring the implications of imprecise probabilities and Choquet integrals.

Contribution

It introduces a generalized utility framework for AIXI that accounts for belief ambiguity and investigates the use of imprecise probabilities and Choquet integrals in expected utility computation.

Findings

Standard recursive value functions are recovered as a special case.

Expected utilities under the death interpretation are not always representable as Choquet integrals.

The paper discusses the computability levels of these generalized expected utilities.

Abstract

We generalize the AIXI reinforcement learning agent to admit a wider class of utility functions. Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history, which is sometimes interpreted as implying a chance of death equal to a quantity called the semimeasure loss. This death interpretation suggests one way to assign utilities to such history prefixes. We argue that it is as natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance. This motivates us to consider the consequences of computing expected utilities with Choquet integrals from imprecise probability theory, including an investigation of their computability level. We recover the standard recursive value function as a special case. However, our most general expected utilities under the death interpretation cannot be characterized as such Choquet integrals.