Reinforcement Learning via AIXI Approximation

TL;DR

This paper presents the first computationally feasible approximation of the AIXI reinforcement learning agent, combining Monte Carlo Tree Search and Context Tree Weighting, demonstrating promising results in complex environments.

Contribution

It introduces a scalable approximation to AIXI, bridging the gap between theory and practical reinforcement learning algorithms.

Findings

Successful approximation of AIXI in complex domains

Effective Monte Carlo Tree Search implementation

Encouraging empirical results in stochastic environments

Abstract

This paper introduces a principled approach for the design of a scalable general reinforcement learning agent. This approach is based on a direct approximation of AIXI, a Bayesian optimality notion for general reinforcement learning agents. Previously, it has been unclear whether the theory of AIXI could motivate the design of practical algorithms. We answer this hitherto open question in the affirmative, by providing the first computationally feasible approximation to the AIXI agent. To develop our approximation, we introduce a Monte Carlo Tree Search algorithm along with an agent-specific extension of the Context Tree Weighting algorithm. Empirically, we present a set of encouraging results on a number of stochastic, unknown, and partially observable domains.