ACT2G: Attention-based Contrastive Learning for Text-to-Gesture Generation

TL;DR

This paper introduces ACT2G, a novel attention-based contrastive learning method for generating content-representative gestures from text, improving realism and diversity in avatar communication.

Contribution

It proposes a new contrastive learning approach that aligns text and gesture features in a shared latent space, enabling content-aware gesture generation.

Findings

User study shows ACT2G outperforms existing methods

Generated gestures better reflect text content

Wide variation in gestures from same text demonstrated

Abstract

Recent increase of remote-work, online meeting and tele-operation task makes people find that gesture for avatars and communication robots is more important than we have thought. It is one of the key factors to achieve smooth and natural communication between humans and AI systems and has been intensively researched. Current gesture generation methods are mostly based on deep neural network using text, audio and other information as the input, however, they generate gestures mainly based on audio, which is called a beat gesture. Although the ratio of the beat gesture is more than 70% of actual human gestures, content based gestures sometimes play an important role to make avatars more realistic and human-like. In this paper, we propose a attention-based contrastive learning for text-to-gesture (ACT2G), where generated gestures represent content of the text by estimating attention weight for each word from the input text. In the method, since text and gesture features calculated by the attention weight are mapped to the same latent space by contrastive learning, once text is given as input, the network outputs a feature vector which can be used to generate gestures related to the content. User study confirmed that the gestures generated by ACT2G were better than existing methods. In addition, it was demonstrated that wide variation of gestures were generated from the same text by changing attention weights by creators.