MOSS: Motion-based 3D Clothed Human Synthesis from Monocular Video

TL;DR

MOSS introduces a motion-aware framework for realistic 3D clothed human reconstruction from monocular videos, leveraging kinematic information to improve surface deformation and visual quality.

Contribution

The paper presents a novel motion-based approach with KGAS and UID modules, explicitly modeling global motion and local deformations for enhanced 3D human synthesis.

Findings

Achieves state-of-the-art visual quality in 3D clothed human synthesis.

Improves Human NeRF and Gaussian Splatting by 33.94% and 16.75% in LPIPS.

Effectively handles local occlusions and surface deformations.

Abstract

Single-view clothed human reconstruction holds a central position in virtual reality applications, especially in contexts involving intricate human motions. It presents notable challenges in achieving realistic clothing deformation. Current methodologies often overlook the influence of motion on surface deformation, resulting in surfaces lacking the constraints imposed by global motion. To overcome these limitations, we introduce an innovative framework, Motion-Based 3D Clo}thed Humans Synthesis (MOSS), which employs kinematic information to achieve motion-aware Gaussian split on the human surface. Our framework consists of two modules: Kinematic Gaussian Locating Splatting (KGAS) and Surface Deformation Detector (UID). KGAS incorporates matrix-Fisher distribution to propagate global motion across the body surface. The density and rotation factors of this distribution explicitly control the Gaussians, thereby enhancing the realism of the reconstructed surface. Additionally, to address local occlusions in single-view, based on KGAS, UID identifies significant surfaces, and geometric reconstruction is performed to compensate for these deformations. Experimental results demonstrate that MOSS achieves state-of-the-art visual quality in 3D clothed human synthesis from monocular videos. Notably, we improve the Human NeRF and the Gaussian Splatting by 33.94% and 16.75% in LPIPS* respectively. Codes are available at https://wanghongsheng01.github.io/MOSS/.