Real-Time Synchronized Interaction Framework for Emotion-Aware Humanoid Robots

TL;DR

This paper introduces a real-time framework for humanoid robots that synchronizes speech and gestures to improve emotional interaction, utilizing a novel dual-channel emotion engine and precise temporal alignment techniques.

Contribution

The framework's integration of a dual-channel emotion engine with dynamic time warping and real-time gesture verification is a novel approach to enhance emotional synchronization in social robots.

Findings

21% higher emotional alignment compared to rule-based systems

Effective coordination of vocal pitch with upper-limb gestures

Maintains lower-body stability during interactions

Abstract

As humanoid robots increasingly introduced into social scene, achieving emotionally synchronized multimodal interaction remains a significant challenges. To facilitate the further adoption and integration of humanoid robots into service roles, we present a real-time framework for NAO robots that synchronizes speech prosody with full-body gestures through three key innovations: (1) A dual-channel emotion engine where large language model (LLM) simultaneously generates context-aware text responses and biomechanically feasible motion descriptors, constrained by a structured joint movement library; (2) Duration-aware dynamic time warping for precise temporal alignment of speech output and kinematic motion keyframes; (3) Closed-loop feasibility verification ensuring gestures adhere to NAO's physical joint limits through real-time adaptation. Evaluations show 21% higher emotional alignment compared to rule-based systems, achieved by coordinating vocal pitch (arousal-driven) with upper-limb kinematics while maintaining lower-body stability. By enabling seamless sensorimotor coordination, this framework advances the deployment of context-aware social robots in dynamic applications such as personalized healthcare, interactive education, and responsive customer service platforms.