Robots Learn Social Skills: End-to-End Learning of Co-Speech Gesture Generation for Humanoid Robots

TL;DR

This paper introduces an end-to-end neural network model that learns to generate human-like co-speech gestures from speech data, improving robot-human interaction without relying on rule-based systems.

Contribution

It presents a novel learning-based approach for co-speech gesture generation from speech data, trained on 52 hours of TED talks, enabling real-time, diverse gesture production.

Findings

Generated gestures were perceived as human-like and contextually appropriate.

The model successfully produced various gesture types including iconic, metaphoric, deictic, and beat gestures.

Real-time gesture generation demonstrated on a NAO robot.

Abstract

Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.