Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots

TL;DR

This paper introduces CMA-ES-IG, an algorithm that enhances user experience in preference-based optimization for assistive robots by generating more intuitive trajectories for user ranking, leading to improved preference learning.

Contribution

The paper presents CMA-ES-IG, a novel trajectory generation algorithm that prioritizes user experience and improves preference learning in assistive robot interactions.

Findings

Users find CMA-ES-IG more intuitive than previous methods.

CMA-ES-IG improves preference learning efficiency.

The algorithm performs well across physical and social robot tasks.

Abstract

Assistive robots interact with humans and must adapt to different users' preferences to be effective. An easy and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, for example, robot movement trajectories or gestures. Existing techniques focus on generating trajectories for users to rank that maximize the outcome of the preference learning process. However, the generated trajectories do not appear to reflect the user's preference over repeated interactions. In this work, we design an algorithm to generate trajectories for users to rank that we call Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG). CMA-ES-IG prioritizes the user's experience of the preference learning process. We show that users find our algorithm more intuitive and easier to use than previous approaches across both physical and social robot tasks. This project's code is hosted at github.com/interaction-lab/CMA-ES-IG