Joint Estimation of Expertise and Reward Preferences From Human Demonstrations

TL;DR

This paper introduces methods for robots to jointly infer human expertise and objectives from demonstrations, improving learning and assistance strategies especially with non-expert users.

Contribution

It proposes two inference approaches for jointly estimating human expertise and objectives from non-optimal demonstrations, enhancing robot adaptability.

Findings

Effective inference of human expertise and objectives demonstrated in simulations.

Approaches outperform baseline methods in explaining human behavior.

Real user data validates the proposed methods' practical utility.

Abstract

When a robot learns from human examples, most approaches assume that the human partner provides examples of optimal behavior. However, there are applications in which the robot learns from non-expert humans. We argue that the robot should learn not only about the human's objectives, but also about their expertise level. The robot could then leverage this joint information to reduce or increase the frequency at which it provides assistance to its human's partner or be more cautious when learning new skills from novice users. Similarly, by taking into account the human's expertise, the robot would also be able of inferring a human's true objectives even when the human's fails to properly demonstrate these objectives due to a lack of expertise. In this paper, we propose to jointly infer the expertise level and objective function of a human given observations of their (possibly) non-optimal demonstrations. Two inference approaches are proposed. In the first approach, inference is done over a finite, discrete set of possible objective functions and expertise levels. In the second approach, the robot optimizes over the space of all possible hypotheses and finds the objective function and expertise level that best explain the observed human behavior. We demonstrate our proposed approaches both in simulation and with real user data.