TRiMM: Transformer-Based Rich Motion Matching for Real-Time multi-modal Interaction in Digital Humans

TL;DR

TRiMM introduces a transformer-based framework for real-time, multi-modal 3D gesture generation in digital humans, combining attention mechanisms, sequence modeling, and gesture retrieval to enable responsive, high-quality co-speech gestures.

Contribution

The paper presents a novel multi-modal framework that achieves real-time 3D gesture synthesis with high accuracy and low latency, addressing limitations of previous methods in speed and long-text comprehension.

Findings

Achieves 120 fps inference speed on consumer GPUs

Maintains 0.15 seconds per-sentence latency

Outperforms state-of-the-art gesture generation methods

Abstract

Large Language Model (LLM)-driven digital humans have sparked a series of recent studies on co-speech gesture generation systems. However, existing approaches struggle with real-time synthesis and long-text comprehension. This paper introduces Transformer-Based Rich Motion Matching (TRiMM), a novel multi-modal framework for real-time 3D gesture generation. Our method incorporates three modules: 1) a cross-modal attention mechanism to achieve precise temporal alignment between speech and gestures; 2) a long-context autoregressive model with a sliding window mechanism for effective sequence modeling; 3) a large-scale gesture matching system that constructs an atomic action library and enables real-time retrieval. Additionally, we develop a lightweight pipeline implemented in the Unreal Engine for experimentation. Our approach achieves real-time inference at 120 fps and maintains a per-sentence latency of 0.15 seconds on consumer-grade GPUs (Geforce RTX3060). Extensive subjective and objective evaluations on the ZEGGS, and BEAT datasets demonstrate that our model outperforms current state-of-the-art methods. TRiMM enhances the speed of co-speech gesture generation while ensuring gesture quality, enabling LLM-driven digital humans to respond to speech in real time and synthesize corresponding gestures. Our code is available at https://github.com/teroon/TRiMM-Transformer-Based-Rich-Motion-Matching