Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modeling

TL;DR

This paper introduces RT, a model-based offline reinforcement learning algorithm that guarantees trajectory reliability by considering historical data, leading to improved policy learning and performance on benchmark tasks.

Contribution

The paper proposes a novel reliability-guaranteed transformer (RT) for MORL that accounts for historical information, providing theoretical guarantees and empirical improvements over existing methods.

Findings

RT effectively filters unreliable trajectories based on cumulative reliability.

RT achieves higher returns compared to state-of-the-art MORL methods.

Theoretical performance guarantees support RT's reliability in policy learning.

Abstract

Model-based offline reinforcement learning (MORL) aims to learn a policy by exploiting a dynamics model derived from an existing dataset. Applying conservative quantification to the dynamics model, most existing works on MORL generate trajectories that approximate the real data distribution to facilitate policy learning by using current information (e.g., the state and action at time step $t$). However, these works neglect the impact of historical information on environmental dynamics, leading to the generation of unreliable trajectories that may not align with the real data distribution. In this paper, we propose a new MORL algorithm \textbf{R}eliability-guaranteed \textbf{T}ransformer (RT), which can eliminate unreliable trajectories by calculating the cumulative reliability of the generated trajectory (i.e., using a weighted variational distance away from the real data). Moreover, by sampling candidate actions with high rewards, RT can efficiently generate high-return trajectories from the existing offline data. We theoretically prove the performance guarantees of RT in policy learning, and empirically demonstrate its effectiveness against state-of-the-art model-based methods on several benchmark tasks.