Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views

TL;DR

This paper introduces a neural rendering system for hand-object interaction that enables novel view synthesis and pose estimation from sparse views, using a two-stage approach with shape and appearance priors and a joint model fitting framework.

Contribution

It extends scene understanding techniques to dynamic hand-object scenarios, proposing a two-stage method with offline prior learning and online interaction modeling.

Findings

Outperforms state-of-the-art methods in experiments

Enables stable contact modeling during interactions

Allows for novel view synthesis of hand-object scenes

Abstract

Hand-object interaction understanding and the barely addressed novel view synthesis are highly desired in the immersive communication, whereas it is challenging due to the high deformation of hand and heavy occlusions between hand and object. In this paper, we propose a neural rendering and pose estimation system for hand-object interaction from sparse views, which can also enable 3D hand-object interaction editing. We share the inspiration from recent scene understanding work that shows a scene specific model built beforehand can significantly improve and unblock vision tasks especially when inputs are sparse, and extend it to the dynamic hand-object interaction scenario and propose to solve the problem in two stages. We first learn the shape and appearance prior knowledge of hands and objects separately with the neural representation at the offline stage. During the online stage, we design a rendering-based joint model fitting framework to understand the dynamic hand-object interaction with the pre-built hand and object models as well as interaction priors, which thereby overcomes penetration and separation issues between hand and object and also enables novel view synthesis. In order to get stable contact during the hand-object interaction process in a sequence, we propose a stable contact loss to make the contact region to be consistent. Experiments demonstrate that our method outperforms the state-of-the-art methods. Code and dataset are available in project webpage https://iscas3dv.github.io/HO-NeRF.