Speech-Gesture GAN: Gesture Generation for Robots and Embodied Agents

TL;DR

This paper introduces a GAN-based framework that generates co-speech gestures for embodied agents from speech text and audio, improving human-agent interaction quality.

Contribution

It presents a novel neural network model that learns to generate natural gestures from speech features, trained on a public dataset of co-speech gestures.

Findings

Objective and subjective evaluations confirm the framework's effectiveness.

The model successfully captures relationships between speech and gestures.

Generated gestures are suitable for robots and embodied agents.

Abstract

Embodied agents, in the form of virtual agents or social robots, are rapidly becoming more widespread. In human-human interactions, humans use nonverbal behaviours to convey their attitudes, feelings, and intentions. Therefore, this capability is also required for embodied agents in order to enhance the quality and effectiveness of their interactions with humans. In this paper, we propose a novel framework that can generate sequences of joint angles from the speech text and speech audio utterances. Based on a conditional Generative Adversarial Network (GAN), our proposed neural network model learns the relationships between the co-speech gestures and both semantic and acoustic features from the speech input. In order to train our neural network model, we employ a public dataset containing co-speech gestures with corresponding speech audio utterances, which were captured from a single male native English speaker. The results from both objective and subjective evaluations demonstrate the efficacy of our gesture-generation framework for Robots and Embodied Agents.