MeGA: Hybrid Mesh-Gaussian Head Avatar for High-Fidelity Rendering and Head Editing

TL;DR

MeGA introduces a hybrid mesh-Gaussian approach for high-fidelity, editable head avatars from multi-view videos, effectively modeling different head components with specialized representations for improved rendering quality.

Contribution

The paper presents a novel hybrid mesh-Gaussian head avatar framework that models facial and hair components with tailored representations, enhancing realism and editing capabilities.

Findings

Outperforms previous state-of-the-art methods in rendering quality.

Supports diverse editing tasks like hairstyle and texture modifications.

Demonstrates effectiveness on the NeRSemble dataset.

Abstract

Creating high-fidelity head avatars from multi-view videos is a core issue for many AR/VR applications. However, existing methods usually struggle to obtain high-quality renderings for all different head components simultaneously since they use one single representation to model components with drastically different characteristics (e.g., skin vs. hair). In this paper, we propose a Hybrid Mesh-Gaussian Head Avatar (MeGA) that models different head components with more suitable representations. Specifically, we select an enhanced FLAME mesh as our facial representation and predict a UV displacement map to provide per-vertex offsets for improved personalized geometric details. To achieve photorealistic renderings, we obtain facial colors using deferred neural rendering and disentangle neural textures into three meaningful parts. For hair modeling, we first build a static canonical hair using 3D Gaussian Splatting. A rigid transformation and an MLP-based deformation field are further applied to handle complex dynamic expressions. Combined with our occlusion-aware blending, MeGA generates higher-fidelity renderings for the whole head and naturally supports more downstream tasks. Experiments on the NeRSemble dataset demonstrate the effectiveness of our designs, outperforming previous state-of-the-art methods and supporting various editing functionalities, including hairstyle alteration and texture editing.