Active Preference-Based Gaussian Process Regression for Reward Learning

TL;DR

This paper introduces an active preference-based Gaussian Process method for reward learning that efficiently infers expressive reward functions from human trajectory preferences, addressing data inefficiency and structural constraints.

Contribution

It proposes a novel active learning framework using Gaussian Processes to learn reward functions solely from human preferences without assuming strict reward structures.

Findings

Efficiently learns reward functions from limited human preferences.

Outperforms existing methods in simulation and user studies.

Handles high-dimensional robotic tasks effectively.

Abstract

Designing reward functions is a challenging problem in AI and robotics. Humans usually have a difficult time directly specifying all the desirable behaviors that a robot needs to optimize. One common approach is to learn reward functions from collected expert demonstrations. However, learning reward functions from demonstrations introduces many challenges: some methods require highly structured models, e.g. reward functions that are linear in some predefined set of features, while others adopt less structured reward functions that on the other hand require tremendous amount of data. In addition, humans tend to have a difficult time providing demonstrations on robots with high degrees of freedom, or even quantifying reward values for given demonstrations. To address these challenges, we present a preference-based learning approach, where as an alternative, the human feedback is only in the form of comparisons between trajectories. Furthermore, we do not assume highly constrained structures on the reward function. Instead, we model the reward function using a Gaussian Process (GP) and propose a mathematical formulation to actively find a GP using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. Our results in simulations and a user study suggest that our approach can efficiently learn expressive reward functions for robotics tasks.