HeadGAN: One-shot Neural Head Synthesis and Editing

TL;DR

HeadGAN is a novel one-shot neural head synthesis system that uses 3D face representations and audio features to improve realism, identity preservation, and expression transfer in head reenactment.

Contribution

It introduces HeadGAN, which conditions synthesis on 3D face representations and audio features, enabling high-quality, identity-preserving head reenactment and editing from a single reference image.

Findings

Achieves improved photo-realism and identity preservation.

Effectively transfers pose and expression from driving videos.

Enables expression and pose editing using 3D face representations.

Abstract

Recent attempts to solve the problem of head reenactment using a single reference image have shown promising results. However, most of them either perform poorly in terms of photo-realism, or fail to meet the identity preservation problem, or do not fully transfer the driving pose and expression. We propose HeadGAN, a novel system that conditions synthesis on 3D face representations, which can be extracted from any driving video and adapted to the facial geometry of any reference image, disentangling identity from expression. We further improve mouth movements, by utilising audio features as a complementary input. The 3D face representation enables HeadGAN to be further used as an efficient method for compression and reconstruction and a tool for expression and pose editing.