Text-to-Video: a Two-stage Framework for Zero-shot Identity-agnostic Talking-head Generation

TL;DR

This paper introduces a two-stage zero-shot framework for identity-agnostic talking-head video generation, combining text-to-speech conversion with audio-driven video synthesis to enable flexible, person-independent video creation from text.

Contribution

It proposes a novel two-stage framework that integrates pretrained zero-shot models for text-to-speech and talking head generation, enabling identity-agnostic video synthesis.

Findings

Compared different TTS and talking head methods to identify the best approach.

Demonstrated the effectiveness of the two-stage framework with sample videos.

Provided a public repository for samples and further research.

Abstract

The advent of ChatGPT has introduced innovative methods for information gathering and analysis. However, the information provided by ChatGPT is limited to text, and the visualization of this information remains constrained. Previous research has explored zero-shot text-to-video (TTV) approaches to transform text into videos. However, these methods lacked control over the identity of the generated audio, i.e., not identity-agnostic, hindering their effectiveness. To address this limitation, we propose a novel two-stage framework for person-agnostic video cloning, specifically focusing on TTV generation. In the first stage, we leverage pretrained zero-shot models to achieve text-to-speech (TTS) conversion. In the second stage, an audio-driven talking head generation method is employed to produce compelling videos privided the audio generated in the first stage. This paper presents a comparative analysis of different TTS and audio-driven talking head generation methods, identifying the most promising approach for future research and development. Some audio and videos samples can be found in the following link: https://github.com/ZhichaoWang970201/Text-to-Video/tree/main.