Animatable and Relightable Gaussians for High-fidelity Human Avatar Modeling

TL;DR

This paper presents Animatable Gaussians, a novel avatar representation combining 2D CNNs, 3D Gaussian splatting, and physically-based rendering to produce high-fidelity, relightable, and dynamically detailed human avatars from RGB videos.

Contribution

It introduces a new Gaussian-based avatar model that integrates template-guided 2D parameterization, StyleGAN-based CNNs, and physical rendering for realistic, animatable, and relightable human avatars.

Findings

Outperforms state-of-the-art methods in avatar quality

Produces lifelike, dynamic, and relightable avatars

Effectively models loose clothing and complex appearances

Abstract

Modeling animatable human avatars from RGB videos is a long-standing and challenging problem. Recent works usually adopt MLP-based neural radiance fields (NeRF) to represent 3D humans, but it remains difficult for pure MLPs to regress pose-dependent garment details. To this end, we introduce Animatable Gaussians, a new avatar representation that leverages powerful 2D CNNs and 3D Gaussian splatting to create high-fidelity avatars. To associate 3D Gaussians with the animatable avatar, we learn a parametric template from the input videos, and then parameterize the template on two front & back canonical Gaussian maps where each pixel represents a 3D Gaussian. The learned template is adaptive to the wearing garments for modeling looser clothes like dresses. Such template-guided 2D parameterization enables us to employ a powerful StyleGAN-based CNN to learn the pose-dependent Gaussian maps for modeling detailed dynamic appearances. Furthermore, we introduce a pose projection strategy for better generalization given novel poses. To tackle the realistic relighting of animatable avatars, we introduce physically-based rendering into the avatar representation for decomposing avatar materials and environment illumination. Overall, our method can create lifelike avatars with dynamic, realistic, generalized and relightable appearances. Experiments show that our method outperforms other state-of-the-art approaches.