StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video

TL;DR

StyleAvatar is a real-time, high-fidelity portrait avatar reconstruction method that combines StyleGAN, compositional representation, and innovative augmentation techniques to enable controllable, photo-realistic video re-animation.

Contribution

It introduces a novel StyleGAN-based framework with compositional and sliding window augmentation methods for fast, high-quality, controllable portrait video generation in real-time.

Findings

Achieves high-quality, photo-realistic portrait videos in real-time

Converges within two hours with high image fidelity

Outperforms existing facial reenactment methods in quality and controllability

Abstract

Face reenactment methods attempt to restore and re-animate portrait videos as realistically as possible. Existing methods face a dilemma in quality versus controllability: 2D GAN-based methods achieve higher image quality but suffer in fine-grained control of facial attributes compared with 3D counterparts. In this work, we propose StyleAvatar, a real-time photo-realistic portrait avatar reconstruction method using StyleGAN-based networks, which can generate high-fidelity portrait avatars with faithful expression control. We expand the capabilities of StyleGAN by introducing a compositional representation and a sliding window augmentation method, which enable faster convergence and improve translation generalization. Specifically, we divide the portrait scenes into three parts for adaptive adjustments: facial region, non-facial foreground region, and the background. Besides, our network leverages the best of UNet, StyleGAN and time coding for video learning, which enables high-quality video generation. Furthermore, a sliding window augmentation method together with a pre-training strategy are proposed to improve translation generalization and training performance, respectively. The proposed network can converge within two hours while ensuring high image quality and a forward rendering time of only 20 milliseconds. Furthermore, we propose a real-time live system, which further pushes research into applications. Results and experiments demonstrate the superiority of our method in terms of image quality, full portrait video generation, and real-time re-animation compared to existing facial reenactment methods. Training and inference code for this paper are at https://github.com/LizhenWangT/StyleAvatar.