Hierarchical Reinforcement Learning with Hindsight

TL;DR

This paper presents a hierarchical reinforcement learning method that combines universal value functions and hindsight learning to improve sample efficiency and enable learning at multiple temporal scales.

Contribution

It introduces a novel approach that automates the learning of temporally extended actions across multiple levels of abstraction in RL.

Findings

Significantly accelerates learning in discrete tasks

Effective in continuous control environments

Enables parallel learning of policies at different time scales

Abstract

Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency when rewards are delayed and sparse. We introduce a solution that enables agents to learn temporally extended actions at multiple levels of abstraction in a sample efficient and automated fashion. Our approach combines universal value functions and hindsight learning, allowing agents to learn policies belonging to different time scales in parallel. We show that our method significantly accelerates learning in a variety of discrete and continuous tasks.