Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control

TL;DR

This paper introduces a novel Euclidean data augmentation method for state-based continuous control in reinforcement learning, leveraging limb configuration features to improve data efficiency and performance.

Contribution

The work proposes using limb configuration features for Euclidean augmentation, addressing limitations of joint-based features, and demonstrates significant improvements in RL data efficiency and performance.

Findings

Euclidean augmentation with limb configurations enhances data efficiency.

Significant performance gains across various control tasks.

Minimal hyperparameter tuning required for effective augmentation.

Abstract

Data augmentation creates new data points by transforming the original ones for a reinforcement learning (RL) agent to learn from, which has been shown to be effective for the objective of improving the data efficiency of RL for continuous control. Prior work towards this objective has been largely restricted to perturbation-based data augmentation where new data points are created by perturbing the original ones, which has been impressively effective for tasks where the RL agent observes control states as images with perturbations including random cropping, shifting, etc. This work focuses on state-based control, where the RL agent can directly observe raw kinematic and task features, and considers an alternative data augmentation applied to these features based on Euclidean symmetries under transformations like rotations. We show that the default state features used in exiting benchmark tasks that are based on joint configurations are not amenable to Euclidean transformations. We therefore advocate using state features based on configurations of the limbs (i.e., the rigid bodies connected by the joints) that instead provide rich augmented data under Euclidean transformations. With minimal hyperparameter tuning, we show this new Euclidean data augmentation strategy significantly improves both data efficiency and asymptotic performance of RL on a wide range of continuous control tasks.