Compositional Planning Using Optimal Option Models

TL;DR

This paper introduces a unified framework for composing option models in planning, enabling multi-level abstraction and efficient subgoal achievement through recursive model composition and dynamic programming.

Contribution

It presents a novel approach to composing option models recursively, generalizing Bellman equations for both intra- and inter-option learning, facilitating multi-level hierarchical planning.

Findings

Enables recursive composition of option models for complex planning tasks

Constructs optimal option models for multiple subgoals simultaneously

Provides rapid progress towards subgoals through compositional planning

Abstract

In this paper we introduce a framework for option model composition. Option models are temporal abstractions that, like macro-operators in classical planning, jump directly from a start state to an end state. Prior work has focused on constructing option models from primitive actions, by intra-option model learning; or on using option models to construct a value function, by inter-option planning. We present a unified view of intra- and inter-option model learning, based on a major generalisation of the Bellman equation. Our fundamental operation is the recursive composition of option models into other option models. This key idea enables compositional planning over many levels of abstraction. We illustrate our framework using a dynamic programming algorithm that simultaneously constructs optimal option models for multiple subgoals, and also searches over those option models to provide rapid progress towards other subgoals.