Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling

TL;DR

This paper demonstrates that sequence modeling methods like Decision Transformer are inherently robust to data corruption in offline reinforcement learning, and introduces RDT, a robust variant with techniques to further improve performance under noisy data.

Contribution

The study reveals the robustness of vanilla sequence models against data corruption and proposes RDT, a new method incorporating robust techniques for improved offline RL performance.

Findings

RDT outperforms prior offline RL methods under data corruption.

Sequence modeling methods like Decision Transformer are inherently robust to noisy data.

RDT maintains high performance even with combined training and testing data perturbations.

Abstract

Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods, particularly when the real-world data is limited. Our study reveals that prior research focusing on adapting predominant offline RL methods based on temporal difference learning still falls short under data corruption when the dataset is limited. In contrast, we discover that vanilla sequence modeling methods, such as Decision Transformer, exhibit robustness against data corruption, even without specialized modifications. To unlock the full potential of sequence modeling, we propose Robust Decision Rransformer (RDT) by incorporating three simple yet effective robust techniques: embedding dropout to improve the model's robustness against erroneous inputs, Gaussian weighted learning to mitigate the effects of corrupted labels, and iterative data correction to eliminate corrupted data from the source. Extensive experiments on MuJoCo, Kitchen, and Adroit tasks demonstrate RDT's superior performance under various data corruption scenarios compared to prior methods. Furthermore, RDT exhibits remarkable robustness in a more challenging setting that combines training-time data corruption with test-time observation perturbations. These results highlight the potential of sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world scenarios. Our code is available at https://github.com/jiawei415/RobustDecisionTransformer.