A$^2$-LLM: An End-to-end Conversational Audio Avatar Large Language Model

TL;DR

A$^2$-LLM is an end-to-end multimodal model that enhances conversational digital humans by jointly reasoning about language, audio, and facial expressions, achieving real-time, emotionally expressive interactions.

Contribution

It introduces A$^2$-LLM, a unified framework for conversational avatars that integrates language, audio prosody, and facial motion, along with the FLAME-QA dataset for training.

Findings

Achieves real-time performance with 500 ms latency and 0.7 RTF.

Generates emotionally rich facial expressions beyond lip-sync.

Outperforms cascaded systems in emotional expressiveness.

Abstract

Developing expressive and responsive conversational digital humans is a cornerstone of next-generation human-computer interaction. While large language models (LLMs) have significantly enhanced dialogue capabilities, most current systems still rely on cascaded architectures that connect independent modules. These pipelines are often plagued by accumulated errors, high latency, and poor real-time performance. Lacking access to the underlying conversational context, these pipelines inherently prioritize rigid lip-sync over emotional depth. To address these challenges, we propose A$^2$-LLM, an end-to-end conversational audio avatar large language model that jointly reasons about language, audio prosody, and 3D facial motion within a unified framework. To facilitate training, we introduce FLAME-QA, a high-quality multimodal dataset designed to align semantic intent with expressive facial dynamics within a QA format. By leveraging deep semantic understanding, A$^2$-LLM generates emotionally rich facial movements beyond simple lip-synchronization. Experimental results demonstrate that our system achieves superior emotional expressiveness while maintaining real-time efficiency (500 ms latency, 0.7 RTF).