Planning with Abstract Learned Models While Learning Transferable Subtasks

TL;DR

This paper presents PALM, an algorithm for hierarchical reinforcement learning that learns abstract, modular models for planning, enabling efficient learning and transferability across related tasks.

Contribution

Introduction of PALM, a framework that learns symbolic, abstract models for hierarchical planning and transfer in reinforcement learning.

Findings

PALM effectively integrates planning and execution.

Models learned are modular and transferable.

Experiments show rapid learning of hierarchical models.

Abstract

We introduce an algorithm for model-based hierarchical reinforcement learning to acquire self-contained transition and reward models suitable for probabilistic planning at multiple levels of abstraction. We call this framework Planning with Abstract Learned Models (PALM). By representing subtasks symbolically using a new formal structure, the lifted abstract Markov decision process (L-AMDP), PALM learns models that are independent and modular. Through our experiments, we show how PALM integrates planning and execution, facilitating a rapid and efficient learning of abstract, hierarchical models. We also demonstrate the increased potential for learned models to be transferred to new and related tasks.