Reusable Options through Gradient-based Meta Learning

TL;DR

This paper introduces a gradient-based meta-learning approach to develop reusable options in hierarchical reinforcement learning, enabling faster adaptation across tasks and outperforming existing methods.

Contribution

It formulates the learning of options as a gradient-based meta-learning problem, addressing shortcomings of prior approaches and improving transferability and learning speed.

Findings

Learned options are transferable and accelerate learning.

Proposed method outperforms existing approaches.

Ablation studies confirm the effectiveness of meta-learning components.

Abstract

Hierarchical methods in reinforcement learning have the potential to reduce the amount of decisions that the agent needs to perform when learning new tasks. However, finding reusable useful temporal abstractions that facilitate fast learning remains a challenging problem. Recently, several deep learning approaches were proposed to learn such temporal abstractions in the form of options in an end-to-end manner. In this work, we point out several shortcomings of these methods and discuss their potential negative consequences. Subsequently, we formulate the desiderata for reusable options and use these to frame the problem of learning options as a gradient-based meta-learning problem. This allows us to formulate an objective that explicitly incentivizes options which allow a higher-level decision maker to adjust in few steps to different tasks. Experimentally, we show that our method is able to learn transferable components which accelerate learning and performs better than existing prior methods developed for this setting. Additionally, we perform ablations to quantify the impact of using gradient-based meta-learning as well as other proposed changes.