Modulated Policy Hierarchies

TL;DR

Modulated Policy Hierarchies (MPH) enable end-to-end learning of hierarchical reinforcement learning controllers from sparse rewards by using modulation signals and multi-scale exploration, outperforming recent baselines on robotics tasks.

Contribution

Introduces MPH, a novel hierarchical RL method that learns from sparse rewards without manual reward shaping or subtask definitions.

Findings

MPH outperforms recent baselines on robotics pushing and stacking tasks.

Bit-vector communication enables skill mixing more efficiently than single-skill selection.

Multi-scale intrinsic motivation facilitates exploration at different hierarchy levels.

Abstract

Solving tasks with sparse rewards is a main challenge in reinforcement learning. While hierarchical controllers are an intuitive approach to this problem, current methods often require manual reward shaping, alternating training phases, or manually defined sub tasks. We introduce modulated policy hierarchies (MPH), that can learn end-to-end to solve tasks from sparse rewards. To achieve this, we study different modulation signals and exploration for hierarchical controllers. Specifically, we find that communicating via bit-vectors is more efficient than selecting one out of multiple skills, as it enables mixing between them. To facilitate exploration, MPH uses its different time scales for temporally extended intrinsic motivation at each level of the hierarchy. We evaluate MPH on the robotics tasks of pushing and sparse block stacking, where it outperforms recent baselines.