A Monte Carlo AIXI Approximation

TL;DR

This paper presents the first computationally feasible Monte Carlo approximation of the AIXI reinforcement learning agent, demonstrating promising results in complex stochastic and partially observable environments.

Contribution

It introduces a novel Monte Carlo Tree Search method and an extension to the Context Tree Weighting algorithm to approximate AIXI practically.

Findings

Encouraging results on stochastic domains

Effective approximation of AIXI in complex environments

New algorithms for scalable general reinforcement learning

Abstract

This paper introduces a principled approach for the design of a scalable general reinforcement learning agent. Our approach is based on a direct approximation of AIXI, a Bayesian optimality notion for general reinforcement learning agents. Previously, it has been unclear whether the theory of AIXI could motivate the design of practical algorithms. We answer this hitherto open question in the affirmative, by providing the first computationally feasible approximation to the AIXI agent. To develop our approximation, we introduce a new Monte-Carlo Tree Search algorithm along with an agent-specific extension to the Context Tree Weighting algorithm. Empirically, we present a set of encouraging results on a variety of stochastic and partially observable domains. We conclude by proposing a number of directions for future research.