Hierarchical reinforcement learning with natural language subgoals

TL;DR

This paper introduces a hierarchical reinforcement learning method that uses natural language data from humans to define sub-goal spaces, enabling more effective learning in complex 3D environments.

Contribution

It presents a novel approach that leverages human natural language data to supervise sub-goal spaces in hierarchical reinforcement learning, improving performance in realistic tasks.

Findings

Outperforms agents cloning expert behavior

Better than HRL without supervised sub-goal space

Effective in 3D embodied environments

Abstract

Hierarchical reinforcement learning has been a compelling approach for achieving goal directed behavior over long sequences of actions. However, it has been challenging to implement in realistic or open-ended environments. A main challenge has been to find the right space of sub-goals over which to instantiate a hierarchy. We present a novel approach where we use data from humans solving these tasks to softly supervise the goal space for a set of long range tasks in a 3D embodied environment. In particular, we use unconstrained natural language to parameterize this space. This has two advantages: first, it is easy to generate this data from naive human participants; second, it is flexible enough to represent a vast range of sub-goals in human-relevant tasks. Our approach outperforms agents that clone expert behavior on these tasks, as well as HRL from scratch without this supervised sub-goal space. Our work presents a novel approach to combining human expert supervision with the benefits and flexibility of reinforcement learning.