OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration

TL;DR

OPRIDE is a novel offline preference-based reinforcement learning algorithm that improves query efficiency through in-dataset exploration and reward overoptimization mitigation, with strong empirical and theoretical results.

Contribution

The paper introduces OPRIDE, a new offline PbRL method that enhances query efficiency using a principled exploration strategy and discount scheduling, addressing exploration and overoptimization issues.

Findings

OPRIDE outperforms prior methods with fewer queries.

Empirical results show strong performance across diverse tasks.

Theoretical guarantees support the algorithm's efficiency.

Abstract

Preference-based reinforcement learning (PbRL) can help avoid sophisticated reward designs and align better with human intentions, showing great promise in various real-world applications. However, obtaining human feedback for preferences can be expensive and time-consuming, which forms a strong barrier for PbRL. In this work, we address the problem of low query efficiency in offline PbRL, pinpointing two primary reasons: inefficient exploration and overoptimization of learned reward functions. In response to these challenges, we propose a novel algorithm, \textbf{O}ffline \textbf{P}b\textbf{R}L via \textbf{I}n-\textbf{D}ataset \textbf{E}xploration (OPRIDE), designed to enhance the query efficiency of offline PbRL. OPRIDE consists of two key features: a principled exploration strategy that maximizes the informativeness of the queries and a discount scheduling mechanism aimed at mitigating overoptimization of the learned reward functions. Through empirical evaluations, we demonstrate that OPRIDE significantly outperforms prior methods, achieving strong performance with notably fewer queries. Moreover, we provide theoretical guarantees of the algorithm's efficiency. Experimental results across various locomotion, manipulation, and navigation tasks underscore the efficacy and versatility of our approach.