POLAR: Preference Optimization and Learning Algorithms for Robotics

TL;DR

POLAR is an open-source MATLAB toolbox that uses preference-based learning to optimize and understand robotic parameters based on human feedback, addressing challenges like lack of clear metrics and limited data in robotic parameter tuning.

Contribution

The paper introduces POLAR, a novel toolbox for preference-based learning in robotics that efficiently explores high-dimensional parameter spaces using subjective human feedback.

Findings

POLAR effectively optimizes robotic behaviors according to human preferences.

The toolbox accurately learns the underlying preference landscape.

Demonstrations show successful application in simulation environments.

Abstract

Parameter tuning for robotic systems is a time-consuming and challenging task that often relies on domain expertise of the human operator. Moreover, existing learning methods are not well suited for parameter tuning for many reasons including: the absence of a clear numerical metric for `good robotic behavior'; limited data due to the reliance on real-world experimental data; and the large search space of parameter combinations. In this work, we present an open-source MATLAB Preference Optimization and Learning Algorithms for Robotics toolbox (POLAR) for systematically exploring high-dimensional parameter spaces using human-in-the-loop preference-based learning. This aim of this toolbox is to systematically and efficiently accomplish one of two objectives: 1) to optimize robotic behaviors for human operator preference; 2) to learn the operator's underlying preference landscape to better understand the relationship between adjustable parameters and operator preference. The POLAR toolbox achieves these objectives using only subjective feedback mechanisms (pairwise preferences, coactive feedback, and ordinal labels) to infer a Bayesian posterior over the underlying reward function dictating the user's preferences. We demonstrate the performance of the toolbox in simulation and present various applications of human-in-the-loop preference-based learning.