Contrastive Learning from Exploratory Actions: Leveraging Natural Interactions for Preference Elicitation

TL;DR

This paper introduces CLEA, a contrastive learning approach that leverages users' exploratory actions to learn meaningful robot behavior features, improving preference elicitation efficiency and quality.

Contribution

The paper presents a novel contrastive learning method that uses exploratory user actions to learn semantically meaningful robot behavior features, reducing labeling effort.

Findings

CLEA features outperform self-supervised features in preference elicitation.

CLEA improves metrics: completeness, simplicity, minimality, explainability.

User exploratory actions provide valuable data for learning behavior features.

Abstract

People have a variety of preferences for how robots behave. To understand and reason about these preferences, robots aim to learn a reward function that describes how aligned robot behaviors are with a user's preferences. Good representations of a robot's behavior can significantly reduce the time and effort required for a user to teach the robot their preferences. Specifying these representations -- what "features" of the robot's behavior matter to users -- remains a difficult problem; Features learned from raw data lack semantic meaning and features learned from user data require users to engage in tedious labeling processes. Our key insight is that users tasked with customizing a robot are intrinsically motivated to produce labels through exploratory search; they explore behaviors that they find interesting and ignore behaviors that are irrelevant. To harness this novel data source of exploratory actions, we propose contrastive learning from exploratory actions (CLEA) to learn trajectory features that are aligned with features that users care about. We learned CLEA features from exploratory actions users performed in an open-ended signal design activity (N=25) with a Kuri robot, and evaluated CLEA features through a second user study with a different set of users (N=42). CLEA features outperformed self-supervised features when eliciting user preferences over four metrics: completeness, simplicity, minimality, and explainability.