Free-Viewpoint RGB-D Human Performance Capture and Rendering

TL;DR

This paper introduces a novel view synthesis framework for realistic free-viewpoint human capture and rendering from a single sparse RGB-D sensor, capable of generalizing to unseen identities and poses with high fidelity.

Contribution

The authors propose a new neural rendering architecture that creates dense feature maps and inpainting for high-quality novel view synthesis without actor-specific models.

Findings

Outperforms prior view synthesis methods in quality and robustness.

Generalizes well to unseen identities, poses, and facial expressions.

Produces detailed and realistic renders even with sparse depth data.

Abstract

Capturing and faithfully rendering photo-realistic humans from novel views is a fundamental problem for AR/VR applications. While prior work has shown impressive performance capture results in laboratory settings, it is non-trivial to achieve casual free-viewpoint human capture and rendering for unseen identities with high fidelity, especially for facial expressions, hands, and clothes. To tackle these challenges we introduce a novel view synthesis framework that generates realistic renders from unseen views of any human captured from a single-view and sparse RGB-D sensor, similar to a low-cost depth camera, and without actor-specific models. We propose an architecture to create dense feature maps in novel views obtained by sphere-based neural rendering, and create complete renders using a global context inpainting model. Additionally, an enhancer network leverages the overall fidelity, even in occluded areas from the original view, producing crisp renders with fine details. We show that our method generates high-quality novel views of synthetic and real human actors given a single-stream, sparse RGB-D input. It generalizes to unseen identities, and new poses and faithfully reconstructs facial expressions. Our approach outperforms prior view synthesis methods and is robust to different levels of depth sparsity.