Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

TL;DR

This paper introduces a neural system for high-quality 4D human-object interaction reconstruction and rendering from sparse RGB inputs, effectively handling occlusions and complex interactions for immersive VR/AR applications.

Contribution

It presents a novel layer-wise scene decoupling and interaction-aware reconstruction approach for detailed geometry and texture recovery in challenging scenarios.

Findings

Achieves high-quality geometry reconstruction under occlusions.

Produces photo-realistic textures in free-viewpoint rendering.

Demonstrates effectiveness on complex human-object interactions.

Abstract

4D reconstruction of human-object interaction is critical for immersive VR/AR experience and human activity understanding. Recent advances still fail to recover fine geometry and texture results from sparse RGB inputs, especially under challenging human-object interactions scenarios. In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams. To deal with complex occlusions raised by human-object interactions, we adopt a layer-wise scene decoupling strategy and perform volumetric reconstruction and neural rendering of the human and object. Specifically, for geometry reconstruction, we propose an interaction-aware human-object capture scheme that jointly considers the human reconstruction and object reconstruction with their correlations. Occlusion-aware human reconstruction and robust human-aware object tracking are proposed for consistent 4D human-object dynamic reconstruction. For neural texture rendering, we propose a layer-wise human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions.