LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency

TL;DR

LEASE introduces a sample-efficient offline preference-based reinforcement learning algorithm that uses a learned transition model and an uncertainty-aware mechanism to reduce human labeling effort while maintaining high performance.

Contribution

The paper proposes LEASE, a novel offline PbRL algorithm that leverages a transition model and confidence-based data selection to improve sample efficiency and reward accuracy.

Findings

Achieves comparable performance with fewer preference labels.

Uses a transition model to generate unlabeled preference data.

Provides theoretical guarantees on policy improvement.

Abstract

Offline preference-based reinforcement learning (PbRL) provides an effective way to overcome the challenges of designing reward and the high costs of online interaction. However, since labeling preference needs real-time human feedback, acquiring sufficient preference labels is challenging. To solve this, this paper proposes a offLine prEference-bAsed RL with high Sample Efficiency (LEASE) algorithm, where a learned transition model is leveraged to generate unlabeled preference data. Considering the pretrained reward model may generate incorrect labels for unlabeled data, we design an uncertainty-aware mechanism to ensure the performance of reward model, where only high confidence and low variance data are selected. Moreover, we provide the generalization bound of reward model to analyze the factors influencing reward accuracy, and demonstrate that the policy learned by LEASE has theoretical improvement guarantee. The developed theory is based on state-action pair, which can be easily combined with other offline algorithms. The experimental results show that LEASE can achieve comparable performance to baseline under fewer preference data without online interaction.