CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

TL;DR

CLARIFY introduces a contrastive learning approach to preference-based reinforcement learning, improving query clarity and embedding meaningful trajectories, especially when human preferences are ambiguous or noisy.

Contribution

It proposes an offline PbRL method that learns a trajectory embedding space to better distinguish preferences and select clearer queries, enhancing label efficiency.

Findings

Outperforms baselines in non-ideal teacher scenarios

Effective with real human feedback

Learns meaningful trajectory embeddings

Abstract

Preference-based reinforcement learning (PbRL) bypasses explicit reward engineering by inferring reward functions from human preference comparisons, enabling better alignment with human intentions. However, humans often struggle to label a clear preference between similar segments, reducing label efficiency and limiting PbRL's real-world applicability. To address this, we propose an offline PbRL method: Contrastive LeArning for ResolvIng Ambiguous Feedback (CLARIFY), which learns a trajectory embedding space that incorporates preference information, ensuring clearly distinguished segments are spaced apart, thus facilitating the selection of more unambiguous queries. Extensive experiments demonstrate that CLARIFY outperforms baselines in both non-ideal teachers and real human feedback settings. Our approach not only selects more distinguished queries but also learns meaningful trajectory embeddings.