MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancement

TL;DR

MANGO is a novel two-stage framework for natural multi-speaker 3D talking head generation that leverages image-level supervision and a new dataset to improve realism and conversational fluidity.

Contribution

The paper introduces MANGO, a two-stage method combining diffusion-based modeling and photometric supervision, and presents MANGO-Dialog, a large dataset for multi-speaker 3D conversational heads.

Findings

Achieves high realism in multi-speaker 3D dialogue modeling

Outperforms existing methods in accuracy and naturalness

Provides a new dataset for multi-person conversational head generation

Abstract

Current audio-driven 3D head generation methods mainly focus on single-speaker scenarios, lacking natural, bidirectional listen-and-speak interaction. Achieving seamless conversational behavior, where speaking and listening states transition fluidly remains a key challenge. Existing 3D conversational avatar approaches rely on error-prone pseudo-3D labels that fail to capture fine-grained facial dynamics. To address these limitations, we introduce a novel two-stage framework MANGO, which leveraging pure image-level supervision by alternately training to mitigate the noise introduced by pseudo-3D labels, thereby achieving better alignment with real-world conversational behaviors. Specifically, in the first stage, a diffusion-based transformer with a dual-audio interaction module models natural 3D motion from multi-speaker audio. In the second stage, we use a fast 3D Gaussian Renderer to generate high-fidelity images and provide 2D-level photometric supervision for the 3D motions through alternate training. Additionally, we introduce MANGO-Dialog, a high-quality dataset with over 50 hours of aligned 2D-3D conversational data across 500+ identities. Extensive experiments demonstrate that our method achieves exceptional accuracy and realism in modeling two-person 3D dialogue motion, significantly advancing the fidelity and controllability of audio-driven talking heads.