Generation of Real-time Robotic Emotional Expressions Learning from Human Demonstration in Mixed Reality

TL;DR

This paper introduces a framework that learns to generate realistic robotic emotional expressions in real-time from human demonstrations in mixed reality, enhancing robot-human interaction capabilities.

Contribution

It presents a novel system that captures human demonstrations in mixed reality and maps them onto robotic behaviors using a flow-matching generative model.

Findings

Effective real-time generation of emotional expressions

Successful mapping from human demonstrations to robot behaviors

Preliminary validation shows promising results

Abstract

Expressive behaviors in robots are critical for effectively conveying their emotional states during interactions with humans. In this work, we present a framework that autonomously generates realistic and diverse robotic emotional expressions based on expert human demonstrations captured in Mixed Reality (MR). Our system enables experts to teleoperate a virtual robot from a first-person perspective, capturing their facial expressions, head movements, and upper-body gestures, and mapping these behaviors onto corresponding robotic components including eyes, ears, neck, and arms. Leveraging a flow-matching-based generative process, our model learns to produce coherent and varied behaviors in real-time in response to moving objects, conditioned explicitly on given emotional states. A preliminary test validated the effectiveness of our approach for generating autonomous expressions.