Matching options to tasks using Option-Indexed Hierarchical Reinforcement Learning

TL;DR

This paper introduces a novel method called Option-Indexed Hierarchical Reinforcement Learning (OI-HRL) that enables zero-shot transfer of pretrained options to new tasks by learning an affinity function between options and environment items, improving efficiency and reuse.

Contribution

The paper proposes a new option indexing approach for hierarchical RL that allows effective reuse of pretrained options across tasks through an affinity function and meta-training, enabling zero-shot generalization.

Findings

OI-HRL achieves performance comparable to oracular baselines.

OI-HRL shows substantial improvements over baseline methods without option reuse.

The approach demonstrates effective zero-shot transfer in simulated environments.

Abstract

The options framework in Hierarchical Reinforcement Learning breaks down overall goals into a combination of options or simpler tasks and associated policies, allowing for abstraction in the action space. Ideally, these options can be reused across different higher-level goals; indeed, such reuse is necessary to realize the vision of a continual learning agent that can effectively leverage its prior experience. Previous approaches have only proposed limited forms of transfer of prelearned options to new task settings. We propose a novel option indexing approach to hierarchical learning (OI-HRL), where we learn an affinity function between options and the items present in the environment. This allows us to effectively reuse a large library of pretrained options, in zero-shot generalization at test time, by restricting goal-directed learning to only those options relevant to the task at hand. We develop a meta-training loop that learns the representations of options and environments over a series of HRL problems, by incorporating feedback about the relevance of retrieved options to the higher-level goal. We evaluate OI-HRL in two simulated settings - the CraftWorld and AI2THOR environments - and show that we achieve performance competitive with oracular baselines, and substantial gains over a baseline that has the entire option pool available for learning the hierarchical policy.