StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation

TL;DR

StyleTalker is a new model that generates realistic, lip-synced talking head videos from a single image and audio, with controllable and independent motion features.

Contribution

It introduces novel components like a contrastive lip-sync discriminator and a disentangled motion space for improved audio-driven talking head synthesis.

Findings

Outperforms state-of-the-art baselines in perceptual quality

Achieves accurate lip synchronization with input audio

Enables independent control of head motion and lip movements

Abstract

We propose StyleTalker, a novel audio-driven talking head generation model that can synthesize a video of a talking person from a single reference image with accurately audio-synced lip shapes, realistic head poses, and eye blinks. Specifically, by leveraging a pretrained image generator and an image encoder, we estimate the latent codes of the talking head video that faithfully reflects the given audio. This is made possible with several newly devised components: 1) A contrastive lip-sync discriminator for accurate lip synchronization, 2) A conditional sequential variational autoencoder that learns the latent motion space disentangled from the lip movements, such that we can independently manipulate the motions and lip movements while preserving the identity. 3) An auto-regressive prior augmented with normalizing flow to learn a complex audio-to-motion multi-modal latent space. Equipped with these components, StyleTalker can generate talking head videos not only in a motion-controllable way when another motion source video is given but also in a completely audio-driven manner by inferring realistic motions from the input audio. Through extensive experiments and user studies, we show that our model is able to synthesize talking head videos with impressive perceptual quality which are accurately lip-synced with the input audios, largely outperforming state-of-the-art baselines.