Goal-conditioned Imitation Learning

TL;DR

This paper introduces a goal-conditioned imitation learning approach that leverages demonstrations to efficiently train policies capable of reaching diverse goals in robotics, reducing sample complexity and handling demonstrations without action data.

Contribution

The work presents a novel method integrating demonstrations into goal-conditioned RL, improving convergence speed and performance, even with actionless trajectories.

Findings

Speeds up policy learning compared to traditional HER.

Effective with demonstrations lacking action information.

Surpasses prior imitation learning methods in goal-reaching tasks.

Abstract

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available at https://sites.google.com/view/goalconditioned-il/.