Rendering Humans from Object-Occluded Monocular Videos

TL;DR

This paper introduces OccNeRF, a neural rendering approach that effectively renders humans in heavily occluded monocular videos by integrating geometry and visibility priors, outperforming existing methods in real-world scenarios.

Contribution

The paper proposes a novel surface-based rendering method, OccNeRF, which incorporates geometry and visibility priors to handle severe occlusions in monocular human rendering.

Findings

Outperforms existing methods on simulated occlusions

Demonstrates superior rendering quality in real-world occluded scenes

Validates effectiveness through extensive experiments

Abstract

3D understanding and rendering of moving humans from monocular videos is a challenging task. Despite recent progress, the task remains difficult in real-world scenarios, where obstacles may block the camera view and cause partial occlusions in the captured videos. Existing methods cannot handle such defects due to two reasons. First, the standard rendering strategy relies on point-point mapping, which could lead to dramatic disparities between the visible and occluded areas of the body. Second, the naive direct regression approach does not consider any feasibility criteria (ie, prior information) for rendering under occlusions. To tackle the above drawbacks, we present OccNeRF, a neural rendering method that achieves better rendering of humans in severely occluded scenes. As direct solutions to the two drawbacks, we propose surface-based rendering by integrating geometry and visibility priors. We validate our method on both simulated and real-world occlusions and demonstrate our method's superiority.