Hindsight Preference Learning for Offline Preference-based Reinforcement Learning

TL;DR

This paper introduces Hindsight Preference Learning (HPL), a method that models human preferences based on future outcomes of trajectories, improving reward estimation in offline preference-based RL.

Contribution

HPL is the first approach to incorporate hindsight information for modeling preferences, enhancing reward signals in offline RL from unlabeled datasets.

Findings

HPL improves reward robustness across multiple domains.

HPL outperforms existing methods in preference-based RL tasks.

HPL effectively utilizes large unlabeled datasets for better policy learning.

Abstract

Offline preference-based reinforcement learning (RL), which focuses on optimizing policies using human preferences between pairs of trajectory segments selected from an offline dataset, has emerged as a practical avenue for RL applications. Existing works rely on extracting step-wise reward signals from trajectory-wise preference annotations, assuming that preferences correlate with the cumulative Markovian rewards. However, such methods fail to capture the holistic perspective of data annotation: Humans often assess the desirability of a sequence of actions by considering the overall outcome rather than the immediate rewards. To address this challenge, we propose to model human preferences using rewards conditioned on future outcomes of the trajectory segments, i.e. the hindsight information. For downstream RL optimization, the reward of each step is calculated by marginalizing over possible future outcomes, the distribution of which is approximated by a variational auto-encoder trained using the offline dataset. Our proposed method, Hindsight Preference Learning (HPL), can facilitate credit assignment by taking full advantage of vast trajectory data available in massive unlabeled datasets. Comprehensive empirical studies demonstrate the benefits of HPL in delivering robust and advantageous rewards across various domains. Our code is publicly released at https://github.com/typoverflow/WiseRL.