SentiAvatar: Towards Expressive and Interactive Digital Humans

TL;DR

SentiAvatar is a framework for creating expressive, interactive 3D digital humans that synchronize speech, gestures, and emotions in real time, leveraging large-scale multimodal data and a novel motion generation architecture.

Contribution

It introduces a new multimodal dialogue dataset, a pre-trained motion foundation model, and an audio-aware motion generation architecture for realistic digital humans.

Findings

Achieved state-of-the-art results on SuSuInterActs and BEATv2 datasets.

Generated 6 seconds of motion in 0.3 seconds with multi-turn streaming.

Produced highly synchronized speech, gestures, and expressions in real time.

Abstract

We present SentiAvatar, a framework for building expressive interactive 3D digital humans, and use it to create SuSu, a virtual character that speaks, gestures, and emotes in real time. Achieving such a system remains challenging, as it requires jointly addressing three key problems: the lack of large-scale, high-quality multimodal data, robust semantic-to-motion mapping, and fine-grained frame-level motion-prosody synchronization. To solve these problems, first, we build SuSuInterActs (21K clips, 37 hours), a dialogue corpus captured via optical motion capture around a single character with synchronized speech, full-body motion, and facial expressions. Second, we pre-train a Motion Foundation Model on 200K+ motion sequences, equipping it with rich action priors that go well beyond the conversation. We then propose an audio-aware plan-then-infill architecture that decouples sentence-level semantic planning from frame-level prosody-driven interpolation, so that generated motions are both semantically appropriate and rhythmically aligned with speech. Experiments show that SentiAvatar achieves state-of-the-art on both SuSuInterActs (R@1 43.64%, nearly 2 times the best baseline) and BEATv2 (FGD 4.941, BC 8.078), producing 6s of output in 0.3s with unlimited multi-turn streaming. The source code, model, and dataset are available at https://sentiavatar.github.io.