Bootstrapping Skills

TL;DR

This paper introduces Learning Skills via Bootstrapping (LSB), a method that iteratively learns and combines simple, parameterized skills to efficiently solve complex MDPs, outperforming monolithic policy approaches.

Contribution

The paper proposes a novel bootstrapping approach for learning and combining simple skills, enabling effective policy learning in large, complex MDPs.

Findings

LSB can learn near-optimal policies through iterative skill bootstrapping.

LSB outperforms monolithic policy methods on complex MDPs.

The approach is compatible with various RL algorithms.

Abstract

The monolithic approach to policy representation in Markov Decision Processes (MDPs) looks for a single policy that can be represented as a function from states to actions. For the monolithic approach to succeed (and this is not always possible), a complex feature representation is often necessary since the policy is a complex object that has to prescribe what actions to take all over the state space. This is especially true in large domains with complicated dynamics. It is also computationally inefficient to both learn and plan in MDPs using a complex monolithic approach. We present a different approach where we restrict the policy space to policies that can be represented as combinations of simpler, parameterized skills---a type of temporally extended action, with a simple policy representation. We introduce Learning Skills via Bootstrapping (LSB) that can use a broad family of Reinforcement Learning (RL) algorithms as a "black box" to iteratively learn parametrized skills. Initially, the learned skills are short-sighted but each iteration of the algorithm allows the skills to bootstrap off one another, improving each skill in the process. We prove that this bootstrapping process returns a near-optimal policy. Furthermore, our experiments demonstrate that LSB can solve MDPs that, given the same representational power, could not be solved by a monolithic approach. Thus, planning with learned skills results in better policies without requiring complex policy representations.