Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards

TL;DR

This paper introduces a dynamics-aware reward learning method for preference-based reinforcement learning, significantly improving sample efficiency and policy performance with fewer human preferences.

Contribution

It proposes a novel approach that combines self-supervised dynamics-aware representations with preference-based reward learning, enhancing efficiency and effectiveness.

Findings

Achieves comparable performance with 10x fewer preference labels.

Improves final policy performance over existing methods.

Demonstrates the effectiveness on quadruped and cheetah environments.

Abstract

Preference-based reinforcement learning (PbRL) aligns a robot behavior with human preferences via a reward function learned from binary feedback over agent behaviors. We show that dynamics-aware reward functions improve the sample efficiency of PbRL by an order of magnitude. In our experiments we iterate between: (1) learning a dynamics-aware state-action representation (z^{sa}) via a self-supervised temporal consistency task, and (2) bootstrapping the preference-based reward function from (z^{sa}), which results in faster policy learning and better final policy performance. For example, on quadruped-walk, walker-walk, and cheetah-run, with 50 preference labels we achieve the same performance as existing approaches with 500 preference labels, and we recover 83\% and 66\% of ground truth reward policy performance versus only 38\% and 21\%. The performance gains demonstrate the benefits of explicitly learning a dynamics-aware reward model. Repo: \texttt{https://github.com/apple/ml-reed}.