Meta-Reinforcement Learning of Structured Exploration Strategies

TL;DR

This paper introduces MAESN, a meta-reinforcement learning algorithm that learns structured exploration strategies from prior tasks, significantly improving exploration efficiency in various simulated robotics tasks.

Contribution

The paper presents MAESN, a novel gradient-based meta-RL method that learns structured exploration strategies, outperforming prior meta-RL and task-agnostic exploration approaches.

Findings

MAESN outperforms prior meta-RL methods in exploration tasks.

Structured stochasticity improves exploration effectiveness.

Method is validated on diverse simulated robotics tasks.

Abstract

Exploration is a fundamental challenge in reinforcement learning (RL). Many of the current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we explore how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm -- model agnostic exploration with structured noise (MAESN) -- to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.