Large Body Language Models

TL;DR

This paper introduces Large Body Language Models (LBLMs) and a novel architecture, LBLM-AVA, which combines Transformer-XL and diffusion models to generate realistic, context-aware gestures for virtual agents using multimodal inputs.

Contribution

The paper presents LBLM-AVA, a new architecture integrating multimodal inputs with advanced components for improved gesture generation in human-computer interaction.

Findings

Achieves 30% reduction in Fréchet Gesture Distance

Improves Fréchet Inception Distance by 25%

State-of-the-art performance in gesture realism

Abstract

As virtual agents become increasingly prevalent in human-computer interaction, generating realistic and contextually appropriate gestures in real-time remains a significant challenge. While neural rendering techniques have made substantial progress with static scripts, their applicability to human-computer interactions remains limited. To address this, we introduce Large Body Language Models (LBLMs) and present LBLM-AVA, a novel LBLM architecture that combines a Transformer-XL large language model with a parallelized diffusion model to generate human-like gestures from multimodal inputs (text, audio, and video). LBLM-AVA incorporates several key components enhancing its gesture generation capabilities, such as multimodal-to-pose embeddings, enhanced sequence-to-sequence mapping with redefined attention mechanisms, a temporal smoothing module for gesture sequence coherence, and an attention-based refinement module for enhanced realism. The model is trained on our large-scale proprietary open-source dataset Allo-AVA. LBLM-AVA achieves state-of-the-art performance in generating lifelike and contextually appropriate gestures with a 30% reduction in Fr\'echet Gesture Distance (FGD), and a 25% improvement in Fr\'echet Inception Distance compared to existing approaches.