Rollout Sampling Approximate Policy Iteration

TL;DR

This paper introduces a more computationally efficient variant of approximate policy iteration for reinforcement learning, using bandit-based sampling to evaluate policies, with demonstrated improvements in standard control tasks.

Contribution

It proposes a new policy iteration method that reduces computational effort by addressing the sampling problem with bandit algorithms, outperforming previous methods.

Findings

Achieves similar performance with less computation

Demonstrates order of magnitude improvement in inverted pendulum

Shows effectiveness in mountain-car domain

Abstract

Several researchers have recently investigated the connection between reinforcement learning and classification. We are motivated by proposals of approximate policy iteration schemes without value functions which focus on policy representation using classifiers and address policy learning as a supervised learning problem. This paper proposes variants of an improved policy iteration scheme which addresses the core sampling problem in evaluating a policy through simulation as a multi-armed bandit machine. The resulting algorithm offers comparable performance to the previous algorithm achieved, however, with significantly less computational effort. An order of magnitude improvement is demonstrated experimentally in two standard reinforcement learning domains: inverted pendulum and mountain-car.