Universal Reinforcement Learning

TL;DR

This paper introduces the active LZ algorithm for universal reinforcement learning, enabling an agent to adaptively minimize long-term costs in unknown environments, with proven convergence under certain conditions and demonstrated effectiveness in a game scenario.

Contribution

The paper presents a novel active LZ algorithm that achieves universal reinforcement learning with proven convergence properties in environments with finite memory.

Findings

The active LZ algorithm converges to the optimal average cost under certain conditions.

Experimental results show the algorithm's effectiveness in the Rock-Paper-Scissors game.

The approach bridges data compression techniques with reinforcement learning for universal control.

Abstract

We consider an agent interacting with an unmodeled environment. At each time, the agent makes an observation, takes an action, and incurs a cost. Its actions can influence future observations and costs. The goal is to minimize the long-term average cost. We propose a novel algorithm, known as the active LZ algorithm, for optimal control based on ideas from the Lempel-Ziv scheme for universal data compression and prediction. We establish that, under the active LZ algorithm, if there exists an integer $K$ such that the future is conditionally independent of the past given a window of $K$ consecutive actions and observations, then the average cost converges to the optimum. Experimental results involving the game of Rock-Paper-Scissors illustrate merits of the algorithm.