Improving through Interaction: Searching Behavioral Representation Spaces with CMA-ES-IG

TL;DR

This paper introduces CMA-ES-IG, an algorithm that improves preference learning in robots by considering user experience, leading to better scalability, robustness, and user satisfaction in identifying preferred behaviors.

Contribution

The paper presents CMA-ES-IG, a novel preference learning algorithm that explicitly incorporates user experience considerations, enhancing scalability and robustness in high-dimensional preference spaces.

Findings

CMA-ES-IG scales effectively to high-dimensional preference spaces.

It maintains computational efficiency for complex problems.

Users prefer CMA-ES-IG over existing methods for behavior ranking.

Abstract

Robots that interact with humans must adapt to individual users' preferences to operate effectively in human-centered environments. An intuitive and effective technique to learn non-expert users' preferences is through rankings of robot behaviors, e.g., trajectories, gestures, or voices. Existing techniques primarily focus on generating queries that optimize preference learning outcomes, such as sample efficiency or final preference estimation accuracy. However, the focus on outcome overlooks key user expectations in the process of providing these rankings, which can negatively impact users' adoption of robotic systems. This work proposes the Covariance Matrix Adaptation Evolution Strategies with Information Gain (CMA-ES-IG) algorithm. CMA-ES-IG explicitly incorporates user experience considerations into the preference learning process by suggesting perceptually distinct and informative trajectories for users to rank. We demonstrate these benefits through both simulated studies and real-robot experiments. CMA-ES-IG, compared to state-of-the-art alternatives, (1) scales more effectively to higher-dimensional preference spaces, (2) maintains computational tractability for high-dimensional problems, (3) is robust to noisy or inconsistent user feedback, and (4) is preferred by non-expert users in identifying their preferred robot behaviors. This project's code is available at github.com/interaction-lab/CMA-ES-IG