EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents

TL;DR

EmbodiedHead is a real-time, high-fidelity talking-head framework that enables seamless listening and speaking avatars for conversational agents using a novel diffusion transformer and a single-stream interface.

Contribution

It introduces a diffusion transformer-based framework with a single-stream interface and a new training scheme for high-quality, real-time conversational avatars.

Findings

Achieves state-of-the-art visual quality and motion fidelity.

Operates in as few as four sampling steps for real-time performance.

Effectively suppresses spurious mouth motion during listening.

Abstract

We present EmbodiedHead, a speech-driven talking-head framework that equips LLMs with real-time visual avatars for conversation. A practical embodied avatar must achieve real-time generation, unified listening-speaking behavior, and high rendered visual quality simultaneously. Our framework couples the first Rectified-Flow Diffusion Transformer (DiT) for this task with a differentiable renderer, enabling diverse, high-fidelity generation in as few as four sampling steps. Prior listening-speaking methods rely on dual-stream audio, introducing an interlocutor look-ahead dependency incompatible with causal user--LLM interaction. We instead adopt a single-stream interface with explicit per-frame listening-speaking state conditioning and a Streaming Audio Scheduler, suppressing spurious mouth motion during listening while enabling seamless turn-taking. A two-stage training scheme of coefficient-space pretraining and joint image-domain refinement further closes the gap between motion-level supervision and rendered quality. Extensive experiments demonstrate state-of-the-art visual quality and motion fidelity in both speaking and listening scenarios.