Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing

TL;DR

Semantic-Human is a novel neural rendering method that jointly encodes semantics, appearance, and geometry to produce photorealistic human images with consistent human parsing across views, even with noisy supervision.

Contribution

It extends neural radiance fields to incorporate semantic information and introduces constraints from SMPL for improved motion and geometry modeling, enabling view-consistent human parsing.

Findings

Achieves accurate 2D semantic labels with noisy pseudo-label supervision.

Demonstrates highly competitive results on ZJU-MoCap dataset.

Enables applications like label denoising, synthesis, and image editing.

Abstract

The neural rendering of humans is a topic of great research significance. However, previous works mostly focus on achieving photorealistic details, neglecting the exploration of human parsing. Additionally, classical semantic work are all limited in their ability to efficiently represent fine results in complex motions. Human parsing is inherently related to radiance reconstruction, as similar appearance and geometry often correspond to similar semantic part. Furthermore, previous works often design a motion field that maps from the observation space to the canonical space, while it tends to exhibit either underfitting or overfitting, resulting in limited generalization. In this paper, we present Semantic-Human, a novel method that achieves both photorealistic details and viewpoint-consistent human parsing for the neural rendering of humans. Specifically, we extend neural radiance fields (NeRF) to jointly encode semantics, appearance and geometry to achieve accurate 2D semantic labels using noisy pseudo-label supervision. Leveraging the inherent consistency and smoothness properties of NeRF, Semantic-Human achieves consistent human parsing in both continuous and novel views. We also introduce constraints derived from the SMPL surface for the motion field and regularization for the recovered volumetric geometry. We have evaluated the model using the ZJU-MoCap dataset, and the obtained highly competitive results demonstrate the effectiveness of our proposed Semantic-Human. We also showcase various compelling applications, including label denoising, label synthesis and image editing, and empirically validate its advantageous properties.