Automatic Goal Generation using Dynamical Distance Learning

TL;DR

This paper introduces a method for automatic goal generation in reinforcement learning using a dynamical distance function, creating curricula that improve sample efficiency in complex tasks.

Contribution

The paper proposes a novel self-supervised approach for goal curriculum generation using a dynamical distance function in multi-goal RL.

Findings

Improved sample efficiency over baseline methods

Effective curriculum generation for robotic tasks

Enhanced learning speed in goal-conditioned environments

Abstract

Reinforcement Learning (RL) agents can learn to solve complex sequential decision making tasks by interacting with the environment. However, sample efficiency remains a major challenge. In the field of multi-goal RL, where agents are required to reach multiple goals to solve complex tasks, improving sample efficiency can be especially challenging. On the other hand, humans or other biological agents learn such tasks in a much more strategic way, following a curriculum where tasks are sampled with increasing difficulty level in order to make gradual and efficient learning progress. In this work, we propose a method for automatic goal generation using a dynamical distance function (DDF) in a self-supervised fashion. DDF is a function which predicts the dynamical distance between any two states within a markov decision process (MDP). With this, we generate a curriculum of goals at the appropriate difficulty level to facilitate efficient learning throughout the training process. We evaluate this approach on several goal-conditioned robotic manipulation and navigation tasks, and show improvements in sample efficiency over a baseline method which only uses random goal sampling.