Multi-Level Discovery of Deep Options

TL;DR

This paper introduces DDO, a scalable policy-gradient method for automatically discovering deep options in hierarchical reinforcement learning, which improves exploration and learning efficiency in complex environments.

Contribution

The paper presents a novel recursive approach for discovering multi-level deep options from demonstrations, enabling scalable hierarchical reinforcement learning.

Findings

Accelerates learning in 4 out of 5 Atari environments

Discovers structure in surgical videos matching expert annotations with 72% accuracy

Effective in multi-level hierarchies through decoupled discovery and control policies

Abstract

Augmenting an agent's control with useful higher-level behaviors called options can greatly reduce the sample complexity of reinforcement learning, but manually designing options is infeasible in high-dimensional and abstract state spaces. While recent work has proposed several techniques for automated option discovery, they do not scale to multi-level hierarchies and to expressive representations such as deep networks. We present Discovery of Deep Options (DDO), a policy-gradient algorithm that discovers parametrized options from a set of demonstration trajectories, and can be used recursively to discover additional levels of the hierarchy. The scalability of our approach to multi-level hierarchies stems from the decoupling of low-level option discovery from high-level meta-control policy learning, facilitated by under-parametrization of the high level. We demonstrate that using the discovered options to augment the action space of Deep Q-Network agents can accelerate learning by guiding exploration in tasks where random actions are unlikely to reach valuable states. We show that DDO is effective in adding options that accelerate learning in 4 out of 5 Atari RAM environments chosen in our experiments. We also show that DDO can discover structure in robot-assisted surgical videos and kinematics that match expert annotation with 72% accuracy.