S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

TL;DR

This paper introduces S4RL, a simple self-supervision method using data augmentation to enhance offline reinforcement learning, leading to improved generalization and performance in real-world robotic tasks and benchmarks.

Contribution

The paper demonstrates that simple state space data augmentations combined with smooth Q-network training significantly outperform existing offline RL algorithms.

Findings

Improved performance on MetaWorld and RoboSuite environments.

Enhanced results on D4RL benchmark datasets.

Effective data augmentation strategies for offline RL.

Abstract

Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment. These algorithms have made it possible to learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories. However, current algorithms overfit to the dataset they are trained on and exhibit poor out-of-distribution generalization to the environment when deployed. In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with a state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self-Supervision technique in RL (S4RL), we significantly improve over the current state-of-the-art algorithms on offline robot learning environments such as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].