TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation

TL;DR

This paper introduces TAVID, a unified framework that jointly generates synchronized interactive faces and speech from text and images, addressing the multimodal nature of human conversation.

Contribution

TAVID is the first system to integrate face and speech generation with bidirectional cross-modal mappers for synchronized audio-visual dialogue.

Findings

Effective in generating realistic talking faces.

Responsive listening head behaviors achieved.

High-quality conversational speech produced.

Abstract

The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conversational systems, recent studies have explored talking or listening head generation as well as conversational speech generation. However, these works are typically studied in isolation, overlooking the multimodal nature of human conversation, which involves tightly coupled audio-visual interactions. In this paper, we introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner. TAVID integrates face and speech generation pipelines through two cross-modal mappers (i.e., a motion mapper and a speaker mapper), which enable bidirectional exchange of complementary information between the audio and visual modalities. We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction fluency, and speech quality. Extensive experiments demonstrate the effectiveness of our approach across all these aspects.