On-line Policy Improvement using Monte-Carlo Search

TL;DR

This paper introduces a Monte-Carlo simulation algorithm for real-time policy improvement in adaptive controllers, demonstrating significant error reduction in backgammon and potential for broader applications.

Contribution

The paper presents a parallelizable Monte-Carlo algorithm for policy improvement that effectively enhances performance across various initial policies.

Findings

Substantial error reduction, up to a factor of 5, in backgammon playing strength.

Effective across a range of initial policies, including neural network-based strategies.

Algorithm is easily parallelizable and applicable to other simulation-based adaptive control tasks.

Abstract

We present a Monte-Carlo simulation algorithm for real-time policy improvement of an adaptive controller. In the Monte-Carlo simulation, the long-term expected reward of each possible action is statistically measured, using the initial policy to make decisions in each step of the simulation. The action maximizing the measured expected reward is then taken, resulting in an improved policy. Our algorithm is easily parallelizable and has been implemented on the IBM SP1 and SP2 parallel-RISC supercomputers. We have obtained promising initial results in applying this algorithm to the domain of backgammon. Results are reported for a wide variety of initial policies, ranging from a random policy to TD-Gammon, an extremely strong multi-layer neural network. In each case, the Monte-Carlo algorithm gives a substantial reduction, by as much as a factor of 5 or more, in the error rate of the base players. The algorithm is also potentially useful in many other adaptive control applications in which it is possible to simulate the environment.