Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method

TL;DR

This paper introduces Object Preference Adaptation (OPA), a method enabling robots to quickly adapt to human feedback in new tasks by updating object-specific preferences in a one-shot manner, without extensive retraining.

Contribution

The paper presents a novel one-shot adaptation method that updates object preferences based on minimal human feedback, improving transferability and efficiency in robot learning.

Findings

OPA successfully adapts to human feedback on physical robots.

The method requires only one human intervention for adaptation.

Training on synthetic data enables effective real-world application.

Abstract

For robots to be effectively deployed in novel environments and tasks, they must be able to understand the feedback expressed by humans during intervention. This can either correct undesirable behavior or indicate additional preferences. Existing methods either require repeated episodes of interactions or assume prior known reward features, which is data-inefficient and can hardly transfer to new tasks. We relax these assumptions by describing human tasks in terms of object-centric sub-tasks and interpreting physical interventions in relation to specific objects. Our method, Object Preference Adaptation (OPA), is composed of two key stages: 1) pre-training a base policy to produce a wide variety of behaviors, and 2) online-updating according to human feedback. The key to our fast, yet simple adaptation is that general interaction dynamics between agents and objects are fixed, and only object-specific preferences are updated. Our adaptation occurs online, requires only one human intervention (one-shot), and produces new behaviors never seen during training. Trained on cheap synthetic data instead of expensive human demonstrations, our policy correctly adapts to human perturbations on realistic tasks on a physical 7DOF robot. Videos, code, and supplementary material are provided.