Learning Reward Functions from Scale Feedback

TL;DR

This paper introduces a novel scale feedback method for robots to learn user preferences more effectively by capturing nuanced feedback through sliders, improving learning efficiency over traditional binary choices.

Contribution

It proposes a probabilistic model for scale feedback and demonstrates its advantages through simulations and user studies, advancing preference learning methods.

Findings

Scale feedback improves learning efficiency in simulations.

User studies show more effective preference learning with scale feedback.

The probabilistic model accurately captures user feedback behavior.

Abstract

Today's robots are increasingly interacting with people and need to efficiently learn inexperienced user's preferences. A common framework is to iteratively query the user about which of two presented robot trajectories they prefer. While this minimizes the users effort, a strict choice does not yield any information on how much one trajectory is preferred. We propose scale feedback, where the user utilizes a slider to give more nuanced information. We introduce a probabilistic model on how users would provide feedback and derive a learning framework for the robot. We demonstrate the performance benefit of slider feedback in simulations, and validate our approach in two user studies suggesting that scale feedback enables more effective learning in practice.