RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians

TL;DR

RMAvatar is a new method for creating photorealistic 3D human avatars from monocular videos, combining explicit mesh geometry with Gaussian splatting to capture both motion and appearance with high realism.

Contribution

It introduces a novel mesh-embedded Gaussian splatting approach with a pose-related rectification module for detailed non-rigid deformations, advancing avatar realism from monocular videos.

Findings

Achieves state-of-the-art rendering quality

Demonstrates superior quantitative performance

Effectively captures complex non-rigid deformations

Abstract

We introduce RMAvatar, a novel human avatar representation with Gaussian splatting embedded on mesh to learn clothed avatar from a monocular video. We utilize the explicit mesh geometry to represent motion and shape of a virtual human and implicit appearance rendering with Gaussian Splatting. Our method consists of two main modules: Gaussian initialization module and Gaussian rectification module. We embed Gaussians into triangular faces and control their motion through the mesh, which ensures low-frequency motion and surface deformation of the avatar. Due to the limitations of LBS formula, the human skeleton is hard to control complex non-rigid transformations. We then design a pose-related Gaussian rectification module to learn fine-detailed non-rigid deformations, further improving the realism and expressiveness of the avatar. We conduct extensive experiments on public datasets, RMAvatar shows state-of-the-art performance on both rendering quality and quantitative evaluations. Please see our project page at https://rm-avatar.github.io.