Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

TL;DR

This paper introduces SRDP, a diffusion policy method with state reconstruction that enhances out-of-distribution generalization in offline RL, demonstrating superior performance on benchmarks and real-world tasks.

Contribution

The paper proposes a novel state reconstruction technique integrated with diffusion policies to improve OOD generalization in offline RL.

Findings

SRDP achieves state-of-the-art results on D4RL benchmarks.

SRDP demonstrates significant improvement in OOD navigation tasks.

Ablation studies confirm the importance of state reconstruction in SRDP.

Abstract

Offline Reinforcement Learning (RL) methods leverage previous experiences to learn better policies than the behavior policy used for data collection. However, they face challenges handling distribution shifts due to the lack of online interaction during training. To this end, we propose a novel method named State Reconstruction for Diffusion Policies (SRDP) that incorporates state reconstruction feature learning in the recent class of diffusion policies to address the problem of out-of-distribution (OOD) generalization. Our method promotes learning of generalizable state representation to alleviate the distribution shift caused by OOD states. To illustrate the OOD generalization and faster convergence of SRDP, we design a novel 2D Multimodal Contextual Bandit environment and realize it on a 6-DoF real-world UR10 robot, as well as in simulation, and compare its performance with prior algorithms. In particular, we show the importance of the proposed state reconstruction via ablation studies. In addition, we assess the performance of our model on standard continuous control benchmarks (D4RL), namely the navigation of an 8-DoF ant and forward locomotion of half-cheetah, hopper, and walker2d, achieving state-of-the-art results. Finally, we demonstrate that our method can achieve 167% improvement over the competing baseline on a sparse continuous control navigation task where various regions of the state space are removed from the offline RL dataset, including the region encapsulating the goal.