MoGA: 3D Generative Avatar Prior for Monocular Gaussian Avatar Reconstruction

TL;DR

MoGA introduces a method to reconstruct high-fidelity 3D Gaussian avatars from a single image by integrating a generative avatar prior with model inversion, improving 3D consistency and realism over previous approaches.

Contribution

The paper proposes a novel approach combining generative avatar models with model inversion for monocular 3D avatar reconstruction, addressing limitations of existing 2D diffusion-based methods.

Findings

Outperforms state-of-the-art techniques in 3D avatar reconstruction.

Ensures 3D consistency and realistic appearance from a single view.

Generates inherently animatable 3D Gaussian avatars.

Abstract

We present MoGA, a novel method to reconstruct high-fidelity 3D Gaussian avatars from a single-view image. The main challenge lies in inferring unseen appearance and geometric details while ensuring 3D consistency and realism. Most previous methods rely on 2D diffusion models to synthesize unseen views; however, these generated views are sparse and inconsistent, resulting in unrealistic 3D artifacts and blurred appearance. To address these limitations, we leverage a generative avatar model, that can generate diverse 3D avatars by sampling deformed Gaussians from a learned prior distribution. Due to limited 3D training data, such a 3D model alone cannot capture all image details of unseen identities. Consequently, we integrate it as a prior, ensuring 3D consistency by projecting input images into its latent space and enforcing additional 3D appearance and geometric constraints. Our novel approach formulates Gaussian avatar creation as model inversion by fitting the generative avatar to synthetic views from 2D diffusion models. The generative avatar provides an initialization for model fitting, enforces 3D regularization, and helps in refining pose. Experiments show that our method surpasses state-of-the-art techniques and generalizes well to real-world scenarios. Our Gaussian avatars are also inherently animatable. For code, see https://zj-dong.github.io/MoGA/.