MonoNHR: Monocular Neural Human Renderer

TL;DR

MonoNHR is a novel monocular neural human rendering method that generates free-viewpoint images of arbitrary humans from a single image, without requiring geometry supervision or seeing the subject during training.

Contribution

It introduces a disentangled 3D geometry and texture feature approach and a Mesh Inpainter module leveraging human structural priors, enabling monocular rendering of unseen subjects.

Findings

Outperforms recent methods on ZJU-MoCap, AIST, and HUMBI datasets.

Capable of rendering unseen humans in a monocular setup.

Operates without geometry supervision or prior subject training.

Abstract

Existing neural human rendering methods struggle with a single image input due to the lack of information in invisible areas and the depth ambiguity of pixels in visible areas. In this regard, we propose Monocular Neural Human Renderer (MonoNHR), a novel approach that renders robust free-viewpoint images of an arbitrary human given only a single image. MonoNHR is the first method that (i) renders human subjects never seen during training in a monocular setup, and (ii) is trained in a weakly-supervised manner without geometry supervision. First, we propose to disentangle 3D geometry and texture features and to condition the texture inference on the 3D geometry features. Second, we introduce a Mesh Inpainter module that inpaints the occluded parts exploiting human structural priors such as symmetry. Experiments on ZJU-MoCap, AIST, and HUMBI datasets show that our approach significantly outperforms the recent methods adapted to the monocular case.