Iterative Policy-Space Expansion in Reinforcement Learning

TL;DR

This paper introduces an iterative policy-space expansion method in reinforcement learning, where an agent progressively refines its policy by starting with broad categories and narrowing down, leading to faster learning in complex tasks like Tetris.

Contribution

The paper proposes a novel reinforcement learning algorithm that gradually refines the policy space without external curricula, inspired by human problem-solving strategies.

Findings

Faster learning rate in Tetris compared to existing algorithms

Effective feature categorization before policy refinement

Demonstrates benefits of iterative policy-space expansion

Abstract

Humans and animals solve a difficult problem much more easily when they are presented with a sequence of problems that starts simple and slowly increases in difficulty. We explore this idea in the context of reinforcement learning. Rather than providing the agent with an externally provided curriculum of progressively more difficult tasks, the agent solves a single task utilizing a decreasingly constrained policy space. The algorithm we propose first learns to categorize features into positive and negative before gradually learning a more refined policy. Experimental results in Tetris demonstrate superior learning rate of our approach when compared to existing algorithms.