Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning

TL;DR

This paper introduces an equivariant data augmentation method for offline reinforcement learning, leveraging learned dynamics models and transformations to improve generalization to out-of-distribution goals.

Contribution

It proposes a novel equivariant data augmentation technique using dynamics models and entropy regularization to enhance offline RL generalization.

Findings

Significant improvement in test performance across environments.

Effective use of transformations to augment offline datasets.

Enhanced generalization to out-of-distribution goals.

Abstract

We present a novel approach to address the challenge of generalization in offline reinforcement learning (RL), where the agent learns from a fixed dataset without any additional interaction with the environment. Specifically, we aim to improve the agent's ability to generalize to out-of-distribution goals. To achieve this, we propose to learn a dynamics model and check if it is equivariant with respect to a fixed type of transformation, namely translations in the state space. We then use an entropy regularizer to increase the equivariant set and augment the dataset with the resulting transformed samples. Finally, we learn a new policy offline based on the augmented dataset, with an off-the-shelf offline RL algorithm. Our experimental results demonstrate that our approach can greatly improve the test performance of the policy on the considered environments.