Neural Rendering of Humans in Novel View and Pose from Monocular Video

TL;DR

This paper presents a novel neural rendering method that synthesizes photo-realistic images of humans in new views and poses from monocular videos, effectively handling unseen poses by integrating multi-frame observations and partial point cloud data.

Contribution

It introduces a new approach combining pose-based latent codes and point cloud features with a temporal transformer to improve human rendering in novel views and poses from monocular videos.

Findings

Outperforms existing methods on ZJU-MoCap dataset

Effectively handles unseen poses and views

Improves detail and structure recovery in rendered humans

Abstract

We introduce a new method that generates photo-realistic humans under novel views and poses given a monocular video as input. Despite the significant progress recently on this topic, with several methods exploring shared canonical neural radiance fields in dynamic scene scenarios, learning a user-controlled model for unseen poses remains a challenging task. To tackle this problem, we introduce an effective method to a) integrate observations across several frames and b) encode the appearance at each individual frame. We accomplish this by utilizing both the human pose that models the body shape as well as point clouds that partially cover the human as input. Our approach simultaneously learns a shared set of latent codes anchored to the human pose among several frames, and an appearance-dependent code anchored to incomplete point clouds generated by each frame and its predicted depth. The former human pose-based code models the shape of the performer whereas the latter point cloud-based code predicts fine-level details and reasons about missing structures at the unseen poses. To further recover non-visible regions in query frames, we employ a temporal transformer to integrate features of points in query frames and tracked body points from automatically-selected key frames. Experiments on various sequences of dynamic humans from different datasets including ZJU-MoCap show that our method significantly outperforms existing approaches under unseen poses and novel views given monocular videos as input.