Preference Elicitation for Offline Reinforcement Learning

TL;DR

This paper introduces Sim-OPRL, an offline preference-based reinforcement learning algorithm that uses a learned environment model to efficiently gather preference feedback, bridging the gap between offline RL and preference-based RL.

Contribution

It proposes a novel offline preference-based RL method leveraging environment models and provides theoretical guarantees on sample complexity.

Findings

Empirical results show improved performance over baseline methods.

Theoretical analysis links sample complexity to data coverage of the optimal policy.

Demonstrates effectiveness in various simulated environments.

Abstract

Applying reinforcement learning (RL) to real-world problems is often made challenging by the inability to interact with the environment and the difficulty of designing reward functions. Offline RL addresses the first challenge by considering access to an offline dataset of environment interactions labeled by the reward function. In contrast, Preference-based RL does not assume access to the reward function and learns it from preferences, but typically requires an online interaction with the environment. We bridge the gap between these frameworks by exploring efficient methods for acquiring preference feedback in a fully offline setup. We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm, which leverages a learned environment model to elicit preference feedback on simulated rollouts. Drawing on insights from both the offline RL and the preference-based RL literature, our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy. We provide theoretical guarantees regarding the sample complexity of our approach, dependent on how well the offline data covers the optimal policy. Finally, we demonstrate the empirical performance of Sim-OPRL in various environments.