Hierarchy through Composition with Linearly Solvable Markov Decision Processes

TL;DR

This paper introduces a novel hierarchical reinforcement learning framework that leverages linearly solvable Markov decision processes to enable concurrent execution of multiple macro-actions, improving scalability and flexibility.

Contribution

It proposes a new control hierarchy based on parallel macro-actions using LMDPs, allowing for deep, compositional task representations in reinforcement learning.

Findings

Enables concurrent execution of macro-actions in RL.

Supports deep hierarchical task representations.

Improves scalability of reinforcement learning architectures.

Abstract

Hierarchical architectures are critical to the scalability of reinforcement learning methods. Current hierarchical frameworks execute actions serially, with macro-actions comprising sequences of primitive actions. We propose a novel alternative to these control hierarchies based on concurrent execution of many actions in parallel. Our scheme uses the concurrent compositionality provided by the linearly solvable Markov decision process (LMDP) framework, which naturally enables a learning agent to draw on several macro-actions simultaneously to solve new tasks. We introduce the Multitask LMDP module, which maintains a parallel distributed representation of tasks and may be stacked to form deep hierarchies abstracted in space and time.