Invariant Transform Experience Replay: Data Augmentation for Deep Reinforcement Learning

TL;DR

This paper introduces Invariant Transform Experience Replay, a data augmentation framework exploiting task symmetries to improve learning efficiency in deep reinforcement learning for robotics.

Contribution

It presents a general framework with two techniques, Kaleidoscope and Goal-augmented Experience Replay, to leverage symmetries for faster RL training.

Findings

Significant speedups in learning rates and success rates in robotic tasks.

Achieved up to 13x speedup in certain tasks.

Successfully deployed policies on real robots.

Abstract

Deep Reinforcement Learning (RL) is a promising approach for adaptive robot control, but its current application to robotics is currently hindered by high sample requirements. To alleviate this issue, we propose to exploit the symmetries present in robotic tasks. Intuitively, symmetries from observed trajectories define transformations that leave the space of feasible RL trajectories invariant and can be used to generate new feasible trajectories, which could be used for training. Based on this data augmentation idea, we formulate a general framework, called Invariant Transform Experience Replay that we present with two techniques: (i) Kaleidoscope Experience Replay exploits reflectional symmetries and (ii) Goal-augmented Experience Replay which takes advantage of lax goal definitions. In the Fetch tasks from OpenAI Gym, our experimental results show significant increases in learning rates and success rates. Particularly, we attain a 13, 3, and 5 times speedup in the pushing, sliding, and pick-and-place tasks respectively in the multi-goal setting. Performance gains are also observed in similar tasks with obstacles and we successfully deployed a trained policy on a real Baxter robot. Our work demonstrates that invariant transformations on RL trajectories are a promising methodology to speed up learning in deep RL.