Talking Head from Speech Audio using a Pre-trained Image Generator

TL;DR

This paper introduces a new approach for generating realistic talking-head videos from speech audio and a single image by leveraging a pre-trained StyleGAN and modeling latent space trajectories.

Contribution

It combines a two-stage training process with a pre-trained StyleGAN to produce high-quality talking-head videos from speech and a single identity image.

Findings

Outperforms recent state-of-the-art methods on standard datasets

Achieves high visual quality as measured by PSNR, SSIM, FID, and LMD

Validated components through ablation experiments

Abstract

We propose a novel method for generating high-resolution videos of talking-heads from speech audio and a single 'identity' image. Our method is based on a convolutional neural network model that incorporates a pre-trained StyleGAN generator. We model each frame as a point in the latent space of StyleGAN so that a video corresponds to a trajectory through the latent space. Training the network is in two stages. The first stage is to model trajectories in the latent space conditioned on speech utterances. To do this, we use an existing encoder to invert the generator, mapping from each video frame into the latent space. We train a recurrent neural network to map from speech utterances to displacements in the latent space of the image generator. These displacements are relative to the back-projection into the latent space of an identity image chosen from the individuals depicted in the training dataset. In the second stage, we improve the visual quality of the generated videos by tuning the image generator on a single image or a short video of any chosen identity. We evaluate our model on standard measures (PSNR, SSIM, FID and LMD) and show that it significantly outperforms recent state-of-the-art methods on one of two commonly used datasets and gives comparable performance on the other. Finally, we report on ablation experiments that validate the components of the model. The code and videos from experiments can be found at https://mohammedalghamdi.github.io/talking-heads-acm-mm