Adaptive Querying for Reward Learning from Human Feedback

TL;DR

This paper introduces an adaptive, iterative method for selecting the most informative human feedback formats and states to efficiently learn safety-related penalty functions in robots, improving learning speed and user alignment.

Contribution

It proposes a novel adaptive feedback selection approach that optimizes both query states and feedback formats, leveraging multiple interaction modes for improved learning efficiency.

Findings

Demonstrates sample efficiency in simulation for avoiding unsafe behaviors.

Shows practical effectiveness through a user study with a physical robot.

Highlights the benefits of adaptive feedback in accelerating learning.

Abstract

Learning from human feedback is a popular approach to train robots to adapt to user preferences and improve safety. Existing approaches typically consider a single querying (interaction) format when seeking human feedback and do not leverage multiple modes of user interaction with a robot. We examine how to learn a penalty function associated with unsafe behaviors using multiple forms of human feedback, by optimizing both the query state and feedback format. Our proposed adaptive feedback selection is an iterative, two-phase approach which first selects critical states for querying, and then uses information gain to select a feedback format for querying across the sampled critical states. The feedback format selection also accounts for the cost and probability of receiving feedback in a certain format. Our experiments in simulation demonstrate the sample efficiency of our approach in learning to avoid undesirable behaviors. The results of our user study with a physical robot highlight the practicality and effectiveness of adaptive feedback selection in seeking informative, user-aligned feedback that accelerate learning. Experiment videos, code and appendices are found on our website: https://tinyurl.com/AFS-learning.