DEGAS: Detailed Expressions on Full-Body Gaussian Avatars

TL;DR

DEGAS introduces a novel 3D Gaussian Splatting-based method for creating full-body avatars with detailed facial expressions, bridging 2D portrait expression spaces with 3D avatar rendering for realistic, controllable animations.

Contribution

The paper presents the first 3DGS-based full-body avatar model with detailed facial expressions driven by a 2D portrait-trained expression latent space.

Findings

Effective reproduction of subtle facial expressions in full-body avatars.

Demonstrated photorealistic rendering with accurate expressions.

Extended to audio-driven animation using 2D talking face data.

Abstract

Although neural rendering has made significant advances in creating lifelike, animatable full-body and head avatars, incorporating detailed expressions into full-body avatars remains largely unexplored. We present DEGAS, the first 3D Gaussian Splatting (3DGS)-based modeling method for full-body avatars with rich facial expressions. Trained on multiview videos of a given subject, our method learns a conditional variational autoencoder that takes both the body motion and facial expression as driving signals to generate Gaussian maps in the UV layout. To drive the facial expressions, instead of the commonly used 3D Morphable Models (3DMMs) in 3D head avatars, we propose to adopt the expression latent space trained solely on 2D portrait images, bridging the gap between 2D talking faces and 3D avatars. Leveraging the rendering capability of 3DGS and the rich expressiveness of the expression latent space, the learned avatars can be reenacted to reproduce photorealistic rendering images with subtle and accurate facial expressions. Experiments on an existing dataset and our newly proposed dataset of full-body talking avatars demonstrate the efficacy of our method. We also propose an audio-driven extension of our method with the help of 2D talking faces, opening new possibilities for interactive AI agents.