NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

TL;DR

NeuralHOFusion is a neural method that enables detailed, photo-realistic volumetric capture and rendering of complex human-object interactions from sparse RGBD inputs, combining neural implicit modeling with layer-wise disentanglement.

Contribution

It introduces a novel neural implicit inference scheme with non-rigid key-volume fusion and a layer-wise human-object texture rendering approach for complex interactions.

Findings

Achieves detailed geometry reconstruction under occlusions.

Produces photo-realistic free-view rendering results.

Demonstrates efficiency and effectiveness in complex scenarios.

Abstract

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralHOFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.