Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design

TL;DR

This paper introduces a new randomized exploration algorithm for preference-based reinforcement learning that improves query efficiency and supports parallel feedback collection, with theoretical guarantees and competitive empirical performance.

Contribution

It proposes a tractable meta-algorithm combining randomized exploration with experimental design, providing regret guarantees and enhanced query efficiency in preference-based RL.

Findings

The method achieves competitive performance with reward-based RL.

It requires fewer preference queries compared to existing approaches.

Parallelization of queries is feasible and effective.

Abstract

We study reinforcement learning from human feedback in general Markov decision processes, where agents learn from trajectory-level preference comparisons. A central challenge in this setting is to design algorithms that select informative preference queries to identify the underlying reward while ensuring theoretical guarantees. We propose a meta-algorithm based on randomized exploration, which avoids the computational challenges associated with optimistic approaches and remains tractable. We establish both regret and last-iterate guarantees under mild reinforcement learning oracle assumptions. To improve query complexity, we introduce and analyze an improved algorithm that collects batches of trajectory pairs and applies optimal experimental design to select informative comparison queries. The batch structure also enables parallelization of preference queries, which is relevant in practical deployment as feedback can be gathered concurrently. Empirical evaluation confirms that the proposed method is competitive with reward-based reinforcement learning while requiring a small number of preference queries.