LightAvatar: Efficient Head Avatar as Dynamic Neural Light Field

TL;DR

LightAvatar introduces a neural light field-based head avatar model that achieves real-time rendering speeds and high image quality from monocular videos, suitable for resource-constrained devices.

Contribution

The paper presents the first neural light field head avatar model that enables fast, high-quality rendering without mesh or volume rendering, using a novel network design and distillation training strategy.

Findings

Achieves 174.1 FPS rendering speed on a consumer GPU.

Outperforms previous methods in image quality both quantitatively and qualitatively.

Operates efficiently without customized optimization.

Abstract

Recent works have shown that neural radiance fields (NeRFs) on top of parametric models have reached SOTA quality to build photorealistic head avatars from a monocular video. However, one major limitation of the NeRF-based avatars is the slow rendering speed due to the dense point sampling of NeRF, preventing them from broader utility on resource-constrained devices. We introduce LightAvatar, the first head avatar model based on neural light fields (NeLFs). LightAvatar renders an image from 3DMM parameters and a camera pose via a single network forward pass, without using mesh or volume rendering. The proposed approach, while being conceptually appealing, poses a significant challenge towards real-time efficiency and training stability. To resolve them, we introduce dedicated network designs to obtain proper representations for the NeLF model and maintain a low FLOPs budget. Meanwhile, we tap into a distillation-based training strategy that uses a pretrained avatar model as teacher to synthesize abundant pseudo data for training. A warping field network is introduced to correct the fitting error in the real data so that the model can learn better. Extensive experiments suggest that our method can achieve new SOTA image quality quantitatively or qualitatively, while being significantly faster than the counterparts, reporting 174.1 FPS (512x512 resolution) on a consumer-grade GPU (RTX3090) with no customized optimization.