Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL

TL;DR

This paper introduces TSRL, a novel offline RL algorithm that leverages time-reversal symmetry in system dynamics to improve data efficiency and performance on small datasets, outperforming existing methods.

Contribution

The paper proposes TDM and TSRL, utilizing fundamental symmetry to enhance offline RL, especially with limited data, and introduces a new reliability measure for out-of-distribution samples.

Findings

TSRL performs well with as little as 1% of original data.

Leveraging symmetry improves representations and out-of-distribution detection.

TSRL outperforms recent offline RL algorithms in small data regimes.

Abstract

Offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets without interacting with the environment. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. Real-world data collection is often expensive and uncontrollable, leading to small and narrowly covered datasets and posing significant challenges for practical deployments of offline RL. In this paper, we provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets. Specifically, we propose a Time-reversal symmetry (T-symmetry) enforced Dynamics Model (TDM), which establishes consistency between a pair of forward and reverse latent dynamics. TDM provides both well-behaved representations for small datasets and a new reliability measure for OOD samples based on compliance with the T-symmetry. These can be readily used to construct a new offline RL algorithm (TSRL) with less conservative policy constraints and a reliable latent space data augmentation procedure. Based on extensive experiments, we find TSRL achieves great performance on small benchmark datasets with as few as 1% of the original samples, which significantly outperforms the recent offline RL algorithms in terms of data efficiency and generalizability.Code is available at: https://github.com/pcheng2/TSRL