In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning

TL;DR

This paper introduces In-Dataset Trajectory Return Regularization (DTR), a method that improves offline preference-based reinforcement learning by reducing reward bias and enhancing policy learning through sequence modeling and ensemble normalization.

Contribution

The paper proposes DTR, a novel regularization technique using sequence modeling and ensemble normalization to address reward bias in offline PbRL, improving policy performance.

Findings

DTR outperforms state-of-the-art baselines on multiple benchmarks.

Ensemble normalization effectively balances reward differentiation and accuracy.

Sequence modeling mitigates reward bias and improves trajectory stitching.

Abstract

Offline preference-based reinforcement learning (PbRL) typically operates in two phases: first, use human preferences to learn a reward model and annotate rewards for a reward-free offline dataset; second, learn a policy by optimizing the learned reward via offline RL. However, accurately modeling step-wise rewards from trajectory-level preference feedback presents inherent challenges. The reward bias introduced, particularly the overestimation of predicted rewards, leads to optimistic trajectory stitching, which undermines the pessimism mechanism critical to the offline RL phase. To address this challenge, we propose In-Dataset Trajectory Return Regularization (DTR) for offline PbRL, which leverages conditional sequence modeling to mitigate the risk of learning inaccurate trajectory stitching under reward bias. Specifically, DTR employs Decision Transformer and TD-Learning to strike a balance between maintaining fidelity to the behavior policy with high in-dataset trajectory returns and selecting optimal actions based on high reward labels. Additionally, we introduce an ensemble normalization technique that effectively integrates multiple reward models, balancing the tradeoff between reward differentiation and accuracy. Empirical evaluations on various benchmarks demonstrate the superiority of DTR over other state-of-the-art baselines.