GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion

TL;DR

This paper introduces GAF, a novel method that uses multi-view diffusion priors and normal maps to reconstruct photorealistic, animatable 3D Gaussian head avatars from monocular videos, outperforming previous methods.

Contribution

The paper presents a new approach combining diffusion models, normal maps, and latent upsampling to improve 3D avatar reconstruction from monocular videos.

Findings

Outperforms previous state-of-the-art in novel view synthesis

Produces higher-fidelity avatar reconstructions from monocular videos

Effectively fills missing regions using multi-view diffusion priors

Abstract

We propose a novel approach for reconstructing animatable 3D Gaussian avatars from monocular videos captured by commodity devices like smartphones. Photorealistic 3D head avatar reconstruction from such recordings is challenging due to limited observations, which leaves unobserved regions under-constrained and can lead to artifacts in novel views. To address this problem, we introduce a multi-view head diffusion model, leveraging its priors to fill in missing regions and ensure view consistency in Gaussian splatting renderings. To enable precise viewpoint control, we use normal maps rendered from FLAME-based head reconstruction, which provides pixel-aligned inductive biases. We also condition the diffusion model on VAE features extracted from the input image to preserve facial identity and appearance details. For Gaussian avatar reconstruction, we distill multi-view diffusion priors by using iteratively denoised images as pseudo-ground truths, effectively mitigating over-saturation issues. To further improve photorealism, we apply latent upsampling priors to refine the denoised latent before decoding it into an image. We evaluate our method on the NeRSemble dataset, showing that GAF outperforms previous state-of-the-art methods in novel view synthesis. Furthermore, we demonstrate higher-fidelity avatar reconstructions from monocular videos captured on commodity devices.